Spring batch, using subwriters with different commit intervals

Question

I have a spring batch job with this situation:

[STEP (commit-interval of 2500)]

• Reader: Do a query in a table and extract a combination of a PRODUCT_ID and Service.
• Processor: -- Queries all registers of Table A and Table B with this combination of Product_ID+Service and correct some fields. Finally, return corrected list of items to Writer.
• Writer: -- I have 3 writers inside it. I group the main list in all items that have a flag marked as true to inser, update, delete. I send the sublist to the correct writer.

My "problem" is that I see that in some cases, I am sending 68000~ items to the writer (with a commit-interval of 2500 and I am worried if in a future this can break the job.

I am searching a solution to limit this size to avoid memory problems but I am not sure if spring batch have options to work this type of situations.

I try making sublist inside my writer to make the commits in the "real" commit-interval, anyway, I can't imagine how to avoid this in the Processor.

Answer 1

The chunk size is based on read items. If the processor creates more items, then Spring Batch cannot control this. It is up to you to handle that.