Multiple files of different data structure formats as input in Spring Batch

Question

Based on my research, I know that Spring Batch provides API to handling many different kinds of data file formats.

But I need clarification on how do we supply multiple files of different format in one chunk / Tasklet.

For that, I know that there is MultiResourceItemReader can process multiple files but AFAIK all the files have to be of the same format and data structure.

So, the question is how can we supply multiple files of different data formats as input in a Tasklet ?

Answer 1

Asoub is right and there is no out-of-the-box Spring Batch reader that "reads it all!". However with just a handful of fairly simple and straight forward classes you can make a java config spring batch application that will go through different files with different file formats.

For one of my applications I had a similar type of use case and I wrote a bunch of fairly simple and straight forward implementations and extensions of the Spring Batch framework to create what I call a "generic" reader. So to answer your question: below you will find the code I used to go through different kind of file formats using spring batch. Obviously below you will find the stripped implementation, but it should get you going in the right direction.

One line is represented by a Record:

public class Record {

    private Object[] columns;

    public void setColumnByIndex(Object candidate, int index) {
        columns[index] = candidate;
    }

    public Object getColumnByIndex(int index){
        return columns[index];
    }

    public void setColumns(Object[] columns) {
        this.columns = columns;
    }
}

Each line contains multiple columns and the columns are separated by a delimiter. It does not matter if file1 contains 10 columns and/or if file2 only contains 3 columns.

The following reader simply maps each line to a record:

@Component
public class GenericReader {

    @Autowired
    private GenericLineMapper genericLineMapper;

    @SuppressWarnings({ "unchecked", "rawtypes" })
    public FlatFileItemReader reader(File file) {
        FlatFileItemReader<Record> reader = new FlatFileItemReader();
        reader.setResource(new FileSystemResource(file));
        reader.setLineMapper((LineMapper) genericLineMapper.defaultLineMapper());
        return reader;
    }
}

The mapper takes a line and converts it to an array of objects:

@Component
public class GenericLineMapper {

    @Autowired
    private ApplicationConfiguration applicationConfiguration;

    @SuppressWarnings({ "unchecked", "rawtypes" })
    public DefaultLineMapper defaultLineMapper() {
        DefaultLineMapper lineMapper = new DefaultLineMapper();
        lineMapper.setLineTokenizer(tokenizer());       
        lineMapper.setFieldSetMapper(new CustomFieldSetMapper());
        return lineMapper;
    }

    private DelimitedLineTokenizer tokenizer() {
        DelimitedLineTokenizer tokenize = new DelimitedLineTokenizer();     
        tokenize.setDelimiter(Character.toString(applicationConfiguration.getDelimiter()));
        tokenize.setQuoteCharacter(applicationConfiguration.getQuote());
        return tokenize;
    }
}

The "magic" of converting the columns to the record happens in the FieldSetMapper:

@Component
public class CustomFieldSetMapper implements FieldSetMapper<Record> {

    @Override
    public Record mapFieldSet(FieldSet fieldSet) throws BindException {
        Record record = new Record();
        Object[] row = new Object[fieldSet.getValues().length];
        for (int i = 0; i < fieldSet.getValues().length; i++) {
            row[i] = fieldSet.getValues()[i];
        }
        record.setColumns(row);
        return record;
    }
}

Using yaml configuration the user provides an input directory and a list of file names and ofcourse the appropriate delimiter and character to quote a column if the column contains the delimiter. Here is an exmple of such a yaml configuration:

@Component
@ConfigurationProperties
public class ApplicationConfiguration {

    private String inputDir;
    private List<String> fileNames;
    private char delimiter;
    private char quote;

    // getters and setters ommitted
}

And then the application.yml:

input-dir: src/main/resources/
file-names: [yourfile1.csv, yourfile2.csv, yourfile3.csv]
delimiter: "|"
quote: "\""

And last but not least, putting it all together:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;
    @Autowired
    public StepBuilderFactory stepBuilderFactory;
    @Autowired
    private GenericReader genericReader;
    @Autowired
    private NoOpWriter noOpWriter;
    @Autowired
    private ApplicationConfiguration applicationConfiguration;

    @Bean
    public Job yourJobName() {
        List<Step> steps = new ArrayList<>();
        applicationConfiguration.getFileNames().forEach(f -> steps.add(loadStep(new File(applicationConfiguration.getInputDir() + f))));

        return jobBuilderFactory.get("yourjobName")                
                .start(createParallelFlow(steps))
                .end()
                .build();
    }

    @SuppressWarnings("unchecked")
    public Step loadStep(File file) {
        return stepBuilderFactory.get("step-" + file.getName())
                .<Record, Record> chunk(10)
                .reader(genericReader.reader(file))
                .writer(noOpWriter)
                .build();
    } 

    private Flow createParallelFlow(List<Step> steps) {
        SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
        // max multithreading = -1, no multithreading = 1, smart size = steps.size()
        taskExecutor.setConcurrencyLimit(1); 

        List<Flow> flows = steps.stream()
                .map(step -> new FlowBuilder<Flow>("flow_" + step.getName()).start(step).build())
                .collect(Collectors.toList());

        return new FlowBuilder<SimpleFlow>("parallelStepsFlow")
                .split(taskExecutor)
                .add(flows.toArray(new Flow[flows.size()]))
                .build();
    }
}

For demonstration purposes you can just put all the classes in one package. The NoOpWriter simply logs the 2nd column of my test files.

@Component
public class NoOpWriter implements ItemWriter<Record> {

    @Override
    public void write(List<? extends Record> items) throws Exception {
        items.forEach(i -> System.out.println(i.getColumnByIndex(1)));      
        // NO - OP
    }
}

Good luck :-)

Answer 2

I don't think there is an out-of-the-box Spring batch reader for multiple input format.

You'll have to build your own. Of course you can reuse already existing FileItemReader as delegates in your custom file reader, and for each file type/format, use the right one.