CODEWITHSUNDEEP
spring-batchspring-integration
Suresh Ram
Suresh Ram

Reputation: 41

Spring Batch Integration - Multiple files as single Message

In the sample https://github.com/ghillert/spring-batch-integration-sample, the file inbound adapter is configured to poll for a directory and once the file is placed in that directory, a FileMessageToJobRequest is constructed and the Spring batch Job is getting launched. So for each file, a new FileMessageToJobRequest is constructed and a new Spring batch Job instance is getting created.

We also want to configure a file inbound adapter to poll for the files but want to process all the files using a single batch job instance. For example, If we place 1000 files in the directory and have max-messages-per-poll to 1000, We want to send the name of the 1000 files as one of the parameters to Spring batch Job instead of calling the Job 1000 times.

Is there a way to send list of files that the File Inbound adapter picked up during its one poll as a single message to the subsequent Spring components?

Thank You,
Regards
Suresh

Upvotes: 0

Views: 1182

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121427

Even if it is a single poll, the inbound-channel-adapter emits messages for each entry.

So, to collect them to the single message you need to use an <aggregator>.

Although with that you have to come up with ReleaseStrategy. Even if you can just use 1 as a correlationKey, there is some issue with releasing the group.

You should agree with that you don't always have 1000 of files there to group them to the single message. So, maybe a TimeoutCountSequenceSizeReleaseStrategy is a good compromise to emit the result after some timeout, even if you don't have enough number of files to complete the group by size.

HTH

UPDATE

You can consider to use group-timeout on the <aggregator> to allow to release groups even if there is no a new message during that period.

In addition the there is an expire-groups-upon-completion option to be sure that you "single" will be cleared and removed after each release. To allow to form a new group for the next poll.

Upvotes: 1

Related Questions