CODEWITHSUNDEEP
javamultithreadingcsvspring-batch
chillworld
chillworld

Reputation: 4277

spring batch multithread same sort in reader as writer

We are experimenting with a spring batch single thread to spring batch multithread.

The setup is rather easy :

So we have changed the reader to a JdbcPagingItemReader and converted the sort from

order by firstname, lastname, id;

to

Map<String, Order> sortConfiguration = new HashMap<>();
sortConfiguration.put("firstname", Order.ASCENDING);
sortConfiguration.put("lastname", Order.ASCENDING);
sortConfiguration.put("id", Order.ASCENDING);

the commit-interval is set to 200.
The batch runs fine, but our csv is completly out of order.
I assumed that spring would write in the file after each commit (and hoped that he write page per page in order), but the disorder is greater then chunks 200 lines.
I got for example line 1, 3 and 5 should be together in a thread and line 2 and 4 in another thread.
Is there any option to preserve the order or is the only way to abandon multithread?

Upvotes: 0

Views: 849

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31600

Multi-threading is incompatible with ordering. If you use a multi-threaded step, items will be read, processed and written in an undefined order. This is mentioned in the Multi-threaded Step section of the reference documentation:

The result of the above configuration is that the Step executes by reading, processing,
and writing each chunk of items (each commit interval) in a separate thread of execution.
Note that this means there is no fixed order for the items to be processed, and a chunk
might contain items that are non-consecutive compared to the single-threaded case.

Upvotes: 1

Related Questions