CODEWITHSUNDEEP
javaspringmultithreadingspring-batch
Aniket
Aniket

Reputation: 13

Using FlatFileItemReader and MultiResourceItemReader with SynchronizedItemStreamReader

Trying to process multiple csv files using MultiResourceItemReader and reading each file using FlatFileItemReader. When trying to process step with task executor, getting two issues:

  1. ReaderNotOpenException. This issue occurs because multiple files being processed in multithreaded environment and there may be situation where one thread tries to read from a csv file which was closed by some other thread. To solve this issue I am thinking to use SynchronizedItemStreamReader as below:
MultiResourceItemReader<activeForecast> resourceItemReader = new MultiResourceItemReader<>();
        resourceItemReader.setResources(DataSyncUtils.getFileName(activeForecastFilePath));
        resourceItemReader.setDelegate(activeForecastReader());
        SynchronizedItemStreamReader<activeForecast> synchronizedItemStreamReader = new SynchronizedItemStreamReader<>();
        synchronizedItemStreamReader.setDelegate(resourceItemReader);
        return synchronizedItemStreamReader;
  1. FlatFileItemReader gives inconsistent results when used with task executor. So to overcome this I am planning to use SynchronizedItemStreamReader as suggested here: Can I use FlatfileItemReader with Taskexecutor?

My question: Do I need to use SynchronizedItemStreamReader with both FlatFileItemReader and MultiResourceItemReader? If yes why and how?

    @Bean("activeForecastitemReader")
    @StepScope
    public MultiResourceItemReader<activeForecast> activeForecastItemReader() {
        MultiResourceItemReader<activeForecast> resourceItemReader = new MultiResourceItemReader<>();
        resourceItemReader.setResources(DataSyncUtils.getFileName(activeForecastFilePath));
        resourceItemReader.setDelegate(activeForecastReader());
        return resourceItemReader;
    }

    public FlatFileItemReader<activeForecast> activeForecastReader() {
        FlatFileItemReader<activeForecast> flatFileItemReader = new FlatFileItemReader<>();
        flatFileItemReader.setName("activeForecast-Reader");
        flatFileItemReader.setLineMapper(activeForecastLineMapper());
        flatFileItemReader.setLinesToSkip(1);
        flatFileItemReader.setSaveState(false);
        return flatFileItemReader;
    }

Upvotes: 0

Views: 805

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31710

The MultiResourceItemReader reads files in sequence. So it does not make sense to me to use it in a multi-threaded step.

If you want to go parallel with multiple threads, I believe a partitioned step is a better option. In this cases, you make each worker thread process a distinct file.

Upvotes: 0

Related Questions