Dynamic chunk size based on step execution duration

Question

I have a very computation based Spring Batch job that is executed with multiple threads. The job might be executed on different computers with different hardware.

I currently configured a chunk size of 1.000.000 but depending on the processor speed and the number of processors, the processing of one chunk might take from 30 seconds up to 1 hour.

Is there any pragmatic way, to adapt the chunk size dynamically based on the runtime of the last StepExecution?

I have something like that in mind:

• Initial chunk size 1000; target runtime per step 5 minutes
• First chunk is processed and the real runtime was 1 minute --> increase chunk size by factor 5
• Set new chunk size is 5000
• Next chunk is processed within 5 minutes; no adaptation of chunk size necessary

Optionally scaling down:

• High workload on the system; real runtime was 10 minutes --> decrease of chunk size by factor 1/2 again
• ...

Is some mechanism like that possible to implement?

Answer 1

It is not possible to re-configure a step dynamically while it is running. You can definitely set the chunk-size dynamically at runtime before starting the step, but not once the step is started and is running. The chunk size will remain the same for all chunks.