CODEWITHSUNDEEP
javaspringspring-batch
zsp
zsp

Reputation: 31

Spring batch reading files with MultiResourceItemReader and using ItemReadListener

Here's the scenario: I have a Spring Batch that reads multiple input files, processes them, and finally generates more output files.

Using FlatFileItemReader and restarting the entire Batch with a cron, I can process the files 1 by 1, however it is not feasible to restart the batch every X seconds just to process the files individually.

PS: I use ItemReadListener to add some properties of the object being read within a jobExecutionContext, which will be used later to validate (and generate, or not, the output file).

However, if I use MultiResourceItemReader to read all the input files without completely restarting the whole context (and the resources), the ItemReadListener overwrites the properties of each object (input file) in the jobExecutionContext, so that we only have data from the last one object present in the array of input files.

Is there any way to use the ItemReadListener for each Resource read inside a MultiResourceItemReader?

Example Reader:

@Bean
    public MultiResourceItemReader<CustomObject> multiResourceItemReader() {
      MultiResourceItemReader<CustomObject> resourceItemReader = new MultiResourceItemReader<CustomObject>();
      resourceItemReader.setResources(resources);
      resourceItemReader.setDelegate(reader());
      return resourceItemReader;
    }

@Bean
    public FlatFileItemReader<CustomObject> reader() {
      FlatFileItemReader<CustomObject> reader = new FlatFileItemReader<CustomObject>();
      reader.setLineMapper(customObjectLineMapper());       
      return reader;
    }

Example Step:

@Bean
    public Step loadInputFiles() {
      return stepBuilderFactory.get("loadInputFiles").<CustomObject, CustomObject>chunk(10)
          .reader(multiResourceItemReader())
          .writer(new NoOpItemWriter())
          .listener(customObjectListener())
          .build();
    }

Example Listener:

public class CustomObjectListener implements ItemReadListener<CustomObject> {

    @Value("#{jobExecution.executionContext}")
    private ExecutionContext executionContext;
    
    @Override
    public void beforeRead() {
    }

    @Override
    public void afterRead(CustomObject item) {
        executionContext.put("customProperty", item.getCustomProperty());
    }

    @Override
    public void onReadError(Exception ex) {
    }
}

Scheduler:

public class Scheduler {

    @Autowired
    JobLauncher jobLauncher;
    
    @Autowired
    Job job;
    
    SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
    
    @Scheduled(fixedDelay = 5000, initialDelay = 5000)
    public void scheduleByFixedRate() throws Exception {
        JobParameters params = new JobParametersBuilder().addString("time", format.format(Calendar.getInstance().getTime()))
                                                         .toJobParameters();
        
        jobLauncher.run(job, params);
}

Upvotes: 0

Views: 1484

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31640

Using FlatFileItemReader and restarting the entire Batch with a cron, I can process the files 1 by 1, however it is not feasible to restart the batch every X seconds just to process the files individually.

That is the very reason I always recommend the job-per-file approach over the single-job-for-all-files-with-MultiResourceItemReader approach, like here or here.

Is there any way to use the ItemReadListener for each Resource read inside a MultiResourceItemReader?

No, because the listener is not aware of the resource the item was read from. This is a limitation of the approach itself, not in Spring Batch. What you can do though is make your items aware of the resource they were read from, by implementing ResourceAware.

Upvotes: 0

Related Questions