CODEWITHSUNDEEP
spring-batch
Petar Tahchiev
Petar Tahchiev

Reputation: 4386

Spring-batch reader for frequently modified source

I'm using spring batch and I want to write a job where I have a JPA reader that selects paginated sets of products from the database. Then I have a processor that will perform some operation on every single product (let's say on product A), but performing this operation on product A the item processor will also process some other products too (like product B, product C, etc.). Then the processor will come to product B because it's in line and is given by the reader. But it has already been processed, so it's actually a waste of time/resources to process it again. How should one actually tackle this - is there a modification aware item reader in spring batch? One solution would be in the item processor to check if the product has already been processed, and only if it hasn't been then process it. However checking if the product has been process is actually very resource consuming.

Upvotes: 0

Views: 241

Answers (1)

Michael Minella
Michael Minella

Reputation: 21463

There are two approaches here that I'd consider:

  1. Adjust what you call an "item" - An item is what is returned from the reader. Depending on the design of things, you may want to build a more complex reader that can include the dependent items and therefore only loop through them once. Obviously this is very dependent upon your specific use case.
  2. Use the Process Indicator pattern - The process indicator pattern is what this is for. As you process items, set a flag in the db indicating that they have been processed. Your reader's query is then configured to only read those that have been processed (filtering those out that were updated via the process phase).

Upvotes: 2

Related Questions