CODEWITHSUNDEEP
spring-batchspring-integration
Wasim
Wasim

Reputation: 1935

Remote chunking with Spring Batch job distribution

I have a technical issue running my Spring batch jobs. The Job simply reading records from the DB (MongoDB) , making some calculations on the record (aggregations) and writing the record result to another table. Reading A , Processing A , writing to record B B is an aggregations of many records of A. I want to use remote chunking to vertically scaling my system causing the processing part be scaled and quick. The problem I face that I need to synchronize the A records so processing them will not conflict when writing the result to B. If I distribute 10 A records to 4 slaves they will conflict when writing the aggregate result to B .

Any idea , how to add synchronizing policy when sending messages from the master to the slaves ?

Thanks in advance ...

Upvotes: 1

Views: 414

Answers (1)

Michael Minella
Michael Minella

Reputation: 21503

If you need to synchronize data like you're describing, I'd recommend not going with remote chunking and using partitioning instead. This would allow you to partition by A and eliminate the synchronization issues you're facing. It would also provide additional throughput as you'd be running one processor per slave (same as in remote chunking).

Upvotes: 1

Related Questions