CODEWITHSUNDEEP
javaspringmultithreadingspring-batchspring-batch-tasklet
miplodder
miplodder

Reputation: 933

Continue processing of a Chunk after few records failed, instead of failing complete Chunk & rollbacking the already processed records

I'm describing the current behavior below,

SkipPolicy is configured to skip 2 records at Job level.

Let's say Thread-1 processed it's Chunks successfully.

Below described is execution of Thread-2,

Now, record 1,2 are rollbacked. Then record 1,2 are picked, re-processed by another Thread and records 4,5 are also picked by another Thread and processed successfully.

I want to avoid this complete failure of Chunk when few records are invalid and skip them up to limit (i.e. 2 records at Job level).

Is their a way to achieve it using Spring Batch capabilities?

Note: I was thinking of custom approach wherein I won't be using SkipPolicy, instead I will have a counter for invalid record (and not throwing an Exception, just increment the counter) at JobExecution level. This approach will work by not failing the Chunk, instead skipping invalid records at Job level up to a limit. But it will make the Spring Batch tables data inconsistent (i.e. all invalid records will be considered good ones). Any views on this approach too?

Upvotes: 0

Views: 36

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31710

The chunk-oriented processing model is an all-or-nothing transactional model for a given chunk, which is viewed as a unit.

The exception to this model is a fault-tolerant step, that is able to scan faulty items when a chunk fails to be processed. Technically, it will reprocess the chunk item by item to determine the faulty one(s), that you can skip with a listener.

I want to avoid this complete failure of Chunk when few records are invalid and skip them up to limit (i.e. 2 records at Job level).

This should be possible by using a fault-tolerant step and configuring the exception to skip and the skip limit: https://docs.spring.io/spring-batch/reference/step/chunk-oriented-processing/configuring-skip.html

Regarding the counter based solution, I am not sure this is a good idea, because you need to make sure the counter is correctly used in a transactional and concurrent setup.

Upvotes: 0

Related Questions