CODEWITHSUNDEEP
spring-batch
Julien
Julien

Reputation: 48

Spring Batch - disable ItemReader cache on processor skip, and process filtered rows again

I have been using Spring Batch for some time already, and today I tried to enhance my batchs to make them more bullet-proofs. So I enabled .faultTolerant and .skipLimit/.skip, but I didn't enable any retry strategy.

Nearly all my batchs have 1 reader (JpaPagingItemReader to read data from my database), 1 processor, and 1 writer to create XML files. These 3 parts are all in one step, and my chunk size is usually around 50.

I noticed two side effects when a skippable exception occurs, and I would like to know if there is a way to change these default behaviors =>

  1. I usually read JPA entities in my reader, send them to my processor where I will change their properties to update my database. But when a skippable exception occurs and the entities are processed again, they become detached entities and they don't match the database anymore because their state was updated in the previous process while the database has been rollbacked. I understand that the results of the reader are cached, and that the documentation says that the result of the ItemReader should stay "idempotent" (not changed). But is there a way to disable the cache of the ItemReader, and to force to load a fresh entity when an element is processed again ? It would suit my use case way more.
  2. I noticed that when an element is filtered (the processor returns null), and then a skippable exception occurs, the filtered elements are not processed again after the rollback. Is there a way to change that ?

I searched a lot on stackoverflow, the spring batch documentation and on google before asking this question, but I couldn't find my answer.

Thanks in advance for your precious help, and thanks a lot to the developers of Spring Batch for this great tool !

Upvotes: 2

Views: 704

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31590

  1. But is there a way to disable the cache of the ItemReader

readerTransactionalQueue is what you are looking for (despite the "queue" in the name which does not apply to your case, but still, this is what disables the cache).

  1. I noticed that when an element is filtered (the processor returns null), and then a skippable exception occurs, the filtered elements are not processed again after the rollback. Is there a way to change that ?

I see no obvious way to change that. This is how it worked since the beginning I think (looking at the last modification date). But this makes sense to me, why would one want to reprocess an item if it was filtered? This item would be re-filtered again anyway, unless the processor is not idempotent which is not recommended.

Upvotes: 3

Related Questions