CODEWITHSUNDEEP
springspring-batch
user1324887
user1324887

Reputation: 672

records being processed twice

We have a spring batch which reach a bunch of data in the reader, processes it and writes it. It all happens as a batch.

I noticed that the processor and writer are going over the same data twice, once as a batch and once individual records.

for ex: writer reads 1000 records, sends 1000 records to the processor, sends 1000 records to the writer.

After this the records gets processed again, individually, but only processor and writer are being called.

We have log statements in all reader, processor, and writer and I can see the logs.

Is there any condition which can make the records being processed individually after they have been processed as a list?

    <batch:job id="feeder-job">
    <batch:step id="regular">
        <tasklet>
            <chunk reader="feeder-reader" processor="feeder-processor"
                writer="feeder-composite-writer" commit-interval="#{stepExecutionContext['query-fetch-size']}"
                skip-limit="1000000">
                <skippable-exception-classes>
                    <include class="java.lang.Exception" />
                    <exclude class="org.apache.solr.client.solrj.SolrServerException"/>
                    <exclude class="org.apache.solr.common.SolrException"/>
                    <exclude class="com.batch.feeder.record.RecordFinderException"/>
                </skippable-exception-classes>
            </chunk>
            <listeners>
                <listener ref="feeder-reader" />
            </listeners>
        </tasklet>
    </batch:step>
</batch:job>

Upvotes: 1

Views: 3145

Answers (1)

Sabir Khan
Sabir Khan

Reputation: 10142

You should well read about a feature before using it. Here you are correct that processing is happening twice only after error occurs.

Basically, you have defined a chunk / step which is fault tolerant to certain specified exceptions Configuring Skip Logic

Your step will not fail till total exception count remains below skip-limitbut on errors, chunk items will be processed twice - one by one, the second time and skipping bad records in second processing.

Upvotes: 0

Related Questions