CODEWITHSUNDEEP
javaspringspring-batch
Oren
Oren

Reputation: 1

Spring batch - write multiple files to multiple destinations

I am new to Spring batch and currently working on a new batch job. My batch job:

  1. Reader: read records from database (i.e employees of 5 company)
  2. Processor: process the records
  3. 5 Writers: filter and write the records into each file (i.e employee from company A will be written into company A file at Path A, employee from company B will be written into company B file at Path B), the last writer will write all the resigned employees from company A to company D.
  4. Tasklet: Save the files into database.

Currently I am using CompositeItemWriter to write 5 files and it worked. 

<bean id="ComapanyWriter" 
    class="org.springframework.batch.item.support.CompositeItemWriter">
  <property name="delegates">
    <list>
      <ref bean="CompanyAWriter" />
      <ref bean="CompanyBWriter" />
      <ref bean="CompanyCWriter" />
      <ref bean="CompanyDWriter" />
      <ref bean="ResignedEmpWriter" />
    </list>
  </property>
</bean>

However, I realized that if the reader retrieved 50k records from database, each writer will loop all the records, search for their respective company employee and write into file, which total is 5 x 50k times and redundant. So, I worried about the performance.

Some solution that I can think of:

  1. tried to remove the written records from the data pool (items) so the next writer will loop lesser data but seems like not allowed to do this? (I got this error java.lang.UnsupportedOperationException)
  2. split the records into 5 list (companyAList, companyBList, etc) for each writer to write but I am not sure how to achieve this
  3. split the records into 5 list in writer and write 5 files using 1 writer, but not sure it is feasible.
  4. Pass company ID as parameter so the job will process and write file dynamically based on the parameter. This method will need to trigger job 4 times for 4 company and it cannot process the resigned employees from all company.
  5. Now I am looking at ClassifierCompositeItemWriter, not sure whether it will help.

Please advice me how can I achieve the above method or how to achieve the batch job goal.

Upvotes: 0

Views: 3881

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31590

ClassifierCompositeItemWriter is the way to go. The Classifier will classify items and pass them to the assigned writer. This way, each writer will receive only items of the class it is assigned to and not loop over all the items of the chunk.

You can find an example of how to use the ClassifierCompositeItemWriter here: https://github.com/spring-projects/spring-batch/blob/master/spring-batch-infrastructure/src/test/java/org/springframework/batch/item/support/ClassifierCompositeItemWriterTests.java

There is a similar question to this one, I'm adding it here for reference: Read flat file and write to multiple writers which will write different objects.

Hope this helps.

Upvotes: 1

Related Questions