CODEWITHSUNDEEP
spring-bootcsvspring-batch
breakingcode
breakingcode

Reputation: 73

Writing in the same files using Spring Batch

I am trying to build the batch application in which I will pick the files from the some folders, filter them using name, pass them to the batch operation using MultiResourceItemReader. Then I will implement my own ItemProcessor to change few rows based on some condition.

My requirement is to write the updated data in the same files I am taking input from, I don't know if we can really do this with Spring Batch.

So basically I can't think of how to implement the ItemWriter here, because I need to write the data to the same file and at the same time to the multiple files.

I guess ClassifierCompositeItemWriter can be used here or MultiResourceItemWriter, I have tried to read about them in different stackoverflow answers, but couldn't find anything related to my requirement.

Can anyone help me to implement this.

Code example would be really helpful.

Thanks

Upvotes: 1

Views: 781

Answers (2)

breakingcode
breakingcode

Reputation: 73

Partitioning can be used here, I don't know if this the right approach, but its working in my case very well.

  1. Firstly I created 3 sets of Partitioners for 3 types of files, using MultiResourcePartitioner. I filtered the files from file system using nio.Files class and fed the collection of files to the MultiResourcePartitioner. It will automatically create the partitioner for every file.
  2. Then I write the Reader, Writer and Processor for every Paritioner. In Reader I dynamically picked the filename from the stepExecutionContext and @StepScope annotation, and in writer I used temporary filename to store the output.
  3. Then finally I created a Tasklet for deleting the original files and renaming the temporary files.

Read the documentation about partioning and parallel processing, you will know if it works in your case [doc]:https://docs.spring.io/spring-batch/docs/current/reference/html/scalability.html#partitioning

Upvotes: 0

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31600

While this should be technically possible, I'm not sure it is a good idea for restartability. What would you do if something goes wrong? The input file would have been overridden with new data and you would lose the original items.

I'm probably missing something here, but I see no reason not to keep the original file, which could be deleted before the new one is renamed with the same name in a final step at the end of the job (after all the processing has been successfully done).

Upvotes: 2

Related Questions