CODEWITHSUNDEEP
javaspringspring-batch
membersound
membersound

Reputation: 86727

How to read multiple files in chunks using Spring-Batch?

I'm using Spring-Batch to read csv files sequentially with MultiResourceItemReader.

I want to create a reader that

Now the problem with MultiResourceItemReader is that it will first read the full file1 in chunks, and when the file is finished, it will continue with file2.

How can I create batch steps that will switch between the files based on the chunksize?

Upvotes: 0

Views: 2972

Answers (2)

H4F
H4F

Reputation: 91

you could use something like

    @Bean
    public MultiResourceItemReader<Company> readerCompany() throws IOException {

        DelimitedLineTokenizer dlt = new DelimitedLineTokenizer();
        dlt.setDelimiter("^");
        dlt.setNames("name", "cui", "code", "euid", "companyState", "address");
        dlt.setStrict(false);

        return new MultiResourceItemReaderBuilder<Company>()
                .name("readerCompany")
                .resources(inputCompanyResources)
                .delegate(new FlatFileItemReaderBuilder<Company>()
                        .name("getCompanyStatusReader")
                        .fieldSetMapper(new FieldSetMapper<Company>() {
                            @Override
                            public Company mapFieldSet(FieldSet fieldSet) throws BindException {
                                return Company.builder()
                                        .name(fieldSet.readString("name"))
                                        .localId(fieldSet.readString("cui"))
                                        .code(fieldSet.readString("code"))
                                        .companyStatus(readCompanyStatuses(fieldSet.readString("companyState")))
                                        .address(fieldSet.readString("address"))
                                        .internationalId(fieldSet.readString("euid"))
                                        .build();
                            }
                        })
                        .linesToSkip(1)
                        .lineTokenizer(dlt)
                        .build())
                .build();
    }

Upvotes: 1

Michael Minella
Michael Minella

Reputation: 21463

You're going to need to create a custom reader to address what you're attempting. You can use the FlatFileItemReader under the hood for the actual file reading, but the logic of reading from two files at once you'll have to orchestrate yourself. Just coding off the top of my head, I'd expect something like this:

public class MultiFileReader implements ItemReader<SomeObject> {

    private List<ItemStreamReader> readers;

    public SomeObject read() {
        SomeObject domainObject = new SomeObject();

        for(ItemStreamReader curReader : readers) {
            domainObject.add(curReader.read());
        }

        return domainObject;
    }
}

Upvotes: 1

Related Questions