CODEWITHSUNDEEP
spring-bootspring-batchspring-annotations
Kush Sahu
Kush Sahu

Reputation: 399

How to handle reader's FlatFileParseException from slave step

I have spring batch job application which parse many csv file containing User details. To parse User details I have a LineMapper which parse from csv file.

So first master step reader reads all file from location and then using Partitioner I have slave steps. All slave steps parallel executed. These slave steps parse each file for User details row by row. After processing it moves processed file to Processed folder.

During testing I kept some wrong value in a file which is causing FlatFileParseException by reader of slave step. Here I would like to move this file to another folder like Failed folder. But I am not able to do this.

How I should write files to Failed folder if any slave step reader fails to parse?

I am using Spring Boot, Spring Batch annotation based

Errors:

FlatFileParseException: Parsing error at line: 10 in resource

JobExecutionException: Partition handler returned an unsuccessful step

Code:

BatchConfiguration.java

@Bean(name = "partitionerJob")
  public Job partitionerJob() 
  throws UnexpectedInputException,           MalformedURLException, ParseException {
    return jobs.get("partitioningJob")
      .start(partitionStep())
      .build();
}


@Bean
public Step partitionStep() 
  throws UnexpectedInputException,     MalformedURLException, ParseException {
    return steps.get("partitionStep")
      .partitioner("slaveStep", partitioner())
     .step(slaveStep())
     .taskExecutor(taskExecutor())
     .build();
}


  @Bean
    public CustomMultiResourcePartitioner partitioner() {
      CustomMultiResourcePartitioner partitioner 
  = new CustomMultiResourcePartitioner();
      Resource[] resources;
      try {
        resources = resoursePatternResolver
         .getResources("file:src/main/resources/input/*.csv");
       } catch (IOException e) {
         throw new RuntimeException("I/O problems when resolving"
          + " the input file pattern.", e);
       }
        partitioner.setResources(resources);
        return partitioner;
     }

@StepScope
@Bean
public FlatFileItemReader<Transaction> itemReader(
  @Value("#{stepExecutionContext[fileName]}") String     filename)
  throws UnexpectedInputException, ParseException {

     return new UserDetailReader(fileName);
}


    @Bean
    @StepScope
    public ItemWriter<Transaction> itemWriter(Marshaller     
   marshaller , 
       @Value("#{stepExecutionContext[opFileName]}")     String 
   filename )
       throws MalformedURLException {
         return new UserDetailWriter(fileName);
      }

@Bean
public Step slaveStep() 
  throws UnexpectedInputException,     MalformedURLException, ParseException {
    return steps.get("slaveStep").<User, User>chunk(5)
  .reader(itemReader(null))
  .writer(itemWriter(marshaller(), null))
  .build();
}

CustomMultiResourcePartitioner.java 

public class CustomMultiResourcePartitioner implements Partitioner {

    @Override
    public Map<String, ExecutionContext> partition(int      gridSize) {
        Map<String, ExecutionContext> map = new      HashMap<>(gridSize);
        int i = 0, k = 1;
        for (Resource resource : resources) {
        ExecutionContext context = new ExecutionContext();
        Assert.state(resource.exists(), "Resource does not     exist: "
              + resource);
        context.putString(keyName, resource.getFilename());
        context.putString("opFileName", 
"output"+k+++".xml");
        map.put(PARTITION_KEY + i, context);
           i++;
       }
       return map;
    }
 }

Thanks for help

Upvotes: 1

Views: 1890

Answers (1)

Niraj Sonawane
Niraj Sonawane

Reputation: 11055

You can use skip & skipLimit attributes while defining your step.

Below is sample configuration. Now your slave Step will not fail until the skipLimit is reached. 

@Bean
public Step slaveStep() 
  throws UnexpectedInputException,     MalformedURLException, ParseException {
    return steps.get("slaveStep").<User, User>chunk(5)
  .reader(itemReader(null))
  .writer(itemWriter(marshaller(), null))  
  .faultTolerant()
  .skip(Exception.class)
  .skipLimit(500)               
  .build();
}

Move the file to Failed folder

You can define StepExecutionListener for your slaveStep. in after step method you can get All FailureExceptions for that step. Using that you can put your logic for moving file

Sample code 

public class MyStepListener implements  StepExecutionListener {


    @Override
    public void beforeStep(StepExecution stepExecution) {

        LOGGER.info("MyStepListener beforeStep "+ stepExecution.getSummary());

    }

    @Override
    public ExitStatus afterStep(StepExecution stepExecution) {
        List<Throwable> failureExceptions = stepExecution.getFailureExceptions();
        // move file from one folder to another
        return null;
    }

}

Upvotes: 1

Related Questions