CODEWITHSUNDEEP
javaspringparallel-processingspring-batch
ninjaturtle
ninjaturtle

Reputation: 541

Chunk Processing Spring batch total size unknown

I am reading data from input source which contains timestamp.My query to the source is based on time range on timestamp.

Lets say time range is 1 minute. My reader reads records in 1 minute range and passes that to processor. The processor can process only 100 records at a time. The reader should keep calling processor with chunk of 100 records until all the records are exhausted for that minute. After that writer should be triggered,

How should I configure spring batch to achieve this ?

Upvotes: 0

Views: 1410

Answers (1)

Steef
Steef

Reputation: 31

Your terminology is not entirely correct with respect to the inner workings of Spring Batch. To process items from a reader through an optional processor to a writer Spring Batch uses chunks. Spring Batch reads your items from your ItemReader until NULL is returned by the reader (which indicates that the input stream is exhausted). It then optionally processes these items by calling an ItemProcessor and finally writes the items using your ItemWriter.

The chunk handling is configured using a chunk size, which means the read and processed items are written in chunks of the given chunk size.

With this in mind regarding your question:

  1. You should configure your reader to read items until your configured time slot is up. You then return NULL to indicate that all is read.
  2. You should set the chunk size to 100 to indicate that the writer should be called every 100 items.

Keep in mind that ending the input stream using NULL return in the reader terminates the job with exit status SUCCESS. Spring Batch will NOT allow you to start this specific Job instance again unless you take specific measures to allow this. So, if there are in fact more items to process you should configure your Job using some kind of JobParametersIncrementer or a dummy timestamp parameter.

Another possibility (and my personal preference) is not returning NULL from the reader but throw some exception, e.g. MoreItemsInInputStreamException or TimeRangeExceededException. This way the Job instance FAILS so the Job operator is aware that more is to be done, which he can do by simply restart the Job with the same parameters.

Steef

Upvotes: 0

Related Questions