CODEWITHSUNDEEP
javaspringspring-batch
Dan Coghlan
Dan Coghlan

Reputation: 583

Spring Batch MultiResourceItemReader Chunk Commit Per Resource

I currently have a Spring Batch job that does the following:

  1. Reads a list of csv files using a MultiResourceItemReader which delegates to a FlatFileItemReader.
  2. Splits each file into chunks and writes each chunk as a JMS message, with each message containing the list of lines in the chunk and the filename of the underlying resource in JSON format.

What I want is for each chunk to only contain lines from a single file resource so that the filename on the JMS message will link up to the corresponding file.

The problem is that when processing of one file resource is complete, the reader will just continue and process the next resource meaning that lines from multiple resource files are being inserted into the same chunk and the filename property will not necessarily match the underlying data in the chunk.

Is there any clean way to prevent the reader from including lines from separate file resources in the same chunk?

EDIT: I believe the solution will require using a custom chunk completion policy to somehow determine if the current item being read is from the same resource as the previous line, not sure how feasible this is though. Any thoughts?.

Upvotes: 0

Views: 704

Answers (1)

Dan Coghlan
Dan Coghlan

Reputation: 583

I changed my implementation to use MultiResourcePartitioner to create a partitioned step per file, everything working now.

Upvotes: 0

Related Questions