Using the same reader in multiple steps in a concurrent job where both steps run concurrently

Question

I have a Spring Batch job that differs only in the ItemWriter but should both use the same reader. I created a base class that both steps extend from, since only the writer is different between the 2 steps. I want to be able to run these steps concurrently using a TaskExecutor with about 3 threads so each step would have it own thread. However, I am unable to use the same reader concurrently. Here is the base class which both classes extend. I have not shown the subclasses since they only differ in the method getProductsWriter() which is used in the BaseProcessingStep shown below. Another thing worth mentioning is that final Product written to the databases are different, hence the need for different writers. One of the products uses a ItemProcessor to add additional information to the standard Product with ProductDetails which is not included in the standard Product object. This Spring Batch Step only updates existing Products with the ProductDetails. The other step only inserts new Products which did not exist and don't yet have ProductDetails available. This is what the update step does when the ProductDetail information later becomes available.

                  @Configuration
                  @RequiredArgsConstructor
                  public abstract class BaseProcessingStep {
                     protected final StepBuilderFactory stepBuilderFactory;


                     @Value("${processing.chunk-size}")
                     protected int chunkSize;

                     @Qualifier("mySqlDataSource")
                     final DataSource mySqlDataSource;


                     @Qualifier("productItemsReader")
                     protected final ProductItemsReader productItemsReader;



                     protected abstract ItemWriter<?> getProductsWriter();

                     @Bean("productsProcessingStep")
                     public Step productsProcessingStep() {
                     return stepBuilderFactory.get("productsProcessingStep")
                     .<Product, Product>chunk(chunkSize)
                     .reader(productItemsReader)
                     .writer(getProductsWriter())
                     .build();
               }
            }

I want to run these 2 steps concurrently so here is how I have setup the code:

                   @Bean
                   public Job job(JobRepository jobRepository) {
                     return new JobBuilder("job", jobRepository)
                            .start(splitFlow())
                            .build()        //builds FlowJobBuilder instance
                           .build();       //builds Job instance
                    }

                    @Bean
                    public Flow splitFlow() {
                      return new FlowBuilder<SimpleFlow>("splitFlow")
                            .split(taskExecutor())
                            .add(flow1(), flow2())
                            .build();
                    }

                    @Bean
                    public Flow flow1() {
                      return new FlowBuilder<SimpleFlow>("flow1")
                      .start(productsUpdateStep())
                      .build();

                    @Bean
                    public Flow flow2() {
                      return new FlowBuilder<SimpleFlow>("flow2")
                      .start(productsInsertStep())
                      .build();

}

How can I use the same reader in both processes? Does subclassing using the BaseProcessingStep the correct way to use the same reader? I have never seen an example using the same reader running concurrent steps. I cannot avoid using the same reader since that the source of the Products information. Please can someone show me how to best solve this issue?

Answer 1

I changed the application to use their own readers and writers