CODEWITHSUNDEEP
javaspringspring-batch
atalame
atalame

Reputation: 3

Spring Batch FlatFileItemWriter does only process data after second run

I have a spring batch that downloads multiple csv files via sftp in step1 and then shall do some processing with it in step2.

Everything works fine, but only when I start the program twice (so that the downloaded files already exist when starting the program).

It seems like the software does not see the files when they are not there at start.

I guess it has something to do with the @Value being read before step1() is executed.


@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
    private static final Logger log = LoggerFactory.getLogger(BillProcessor.class);
    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job mainJob() throws IOException, InterruptedException {
        return jobBuilderFactory
                .get("mainJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .next(step2())
                .build();
    }

    @Bean
    protected Step step1() {
        return stepBuilderFactory
                .get("step1")
                .tasklet(sftpTransferer())
                .build();
    }

    @Bean
    public Step step2() throws IOException, InterruptedException {
        return stepBuilderFactory.get("step2").<Bill, Bill>chunk(1000)
                .reader(multiResourceItemReader())
                .writer(writer())
                .build();
    }

    @Bean
    public SftpTransferer sftpTransferer() {
        SftpTransferer tasklet = new SftpTransferer();
        return tasklet;
    }

    @Bean
    public FlatFileItemReader<Bill> reader() throws IOException, InterruptedException {
        log.info("start reader");
        ...
        return reader;
    }

    @Value("file:*.tsv")
    private Resource[] inputResources;

    @Bean
    public MultiResourceItemReader<Bill> multiResourceItemReader() throws InterruptedException, IOException {
        log.info("start MultiResourceItemReader");
        MultiResourceItemReader<Bill> resourceItemReader = new MultiResourceItemReader<Bill>();
        resourceItemReader.setResources(inputResources);
        resourceItemReader.setDelegate(reader());
        return resourceItemReader;
    }

    @Bean
    public BillProcessor processor() {
        log.info("start processor");
        return new BillProcessor();
    }


    @Bean
    public FlatFileItemWriter<Bill> writer() throws IOException {
        log.info("start writer");

        FlatFileItemWriter<Bill> writer = new FlatFileItemWriter<>();
        writer.setResource(new FileSystemResource("_accounting.csv"));
        log.info("set append to true");
        writer.setAppendAllowed(true);
        HeaderWriter headerWriter = new HeaderWriter("id;path;processed;records");
        writer.setHeaderCallback(headerWriter);

        writer.setLineAggregator(new DelimitedLineAggregator<Bill>() {{
            setDelimiter(";");
            setFieldExtractor(new BeanWrapperFieldExtractor<Bill>() {{
                setNames(new String[]{"id", "path", "processed", "records"});
            }});
        }});
        return writer;
    }
}

I could not find anything on the web. Any help appreciated!

Upvotes: 0

Views: 224

Answers (1)

Stefan Reisner
Stefan Reisner

Reputation: 705

I suggest making bean multiresourceItemReader() step-scoped and let it receive the inputResources as an argument:

    @Bean
    public Step step2() throws IOException, InterruptedException {
        return stepBuilderFactory.get("step2").<Bill, Bill>chunk(1000)
                .reader(multiResourceItemReader(null))
                .writer(writer())
                .build();
    }

...

    @StepScope
    @Bean
    public MultiResourceItemReader<Bill> multiResourceItemReader(@Value("file:*.tsv") Resource[] inputResources) throws InterruptedException, IOException {
        log.info("start MultiResourceItemReader");
        MultiResourceItemReader<Bill> resourceItemReader = new MultiResourceItemReader<Bill>();
        resourceItemReader.setResources(inputResources);
        resourceItemReader.setDelegate(reader());
        return resourceItemReader;
    }

This way, collecting the inputResources is deferred until after step1 has been executed, and the files actually exist.

Upvotes: 1

Related Questions