Reputation: 1935
I have a technical issue running my Spring batch jobs. The Job simply reading records from the DB (MongoDB) , making some calculations on the record (aggregations) and writing the record result to another table. Reading A , Processing A , writing to record B B is an aggregations of many records of A. I want to use remote chunking to vertically scaling my system causing the processing part be scaled and quick. The problem I face that I need to synchronize the A records so processing them will not conflict when writing the result to B. If I distribute 10 A records to 4 slaves they will conflict when writing the aggregate result to B .
Any idea , how to add synchronizing policy when sending messages from the master to the slaves ?
Thanks in advance ...
Upvotes: 1
Views: 414
Reputation: 21503
If you need to synchronize data like you're describing, I'd recommend not going with remote chunking and using partitioning instead. This would allow you to partition by A and eliminate the synchronization issues you're facing. It would also provide additional throughput as you'd be running one processor per slave (same as in remote chunking).
Upvotes: 1