CODEWITHSUNDEEP
javaspring-batch
nanpakal
nanpakal

Reputation: 1021

spring batch read from multiple files and write to one file

I have spring batch configuration which reads from multiple files and write mutiple file. Is it possible to write to only one file reading from multiple. Lets say i receive huge XML file, i split XML into small files and use partitioner and read small files parallel. But i need to write all the data read from different small xml files to one output file. Is this possible with spring batch?I know it is possible by making writer synchronized, but i am looking for any other possible way Job configuration

@Bean
    public Job job(final Step parser) {
        return jobBuilderFactory.get("JOB")
                .flow(parser)
                .end()
                .build();
    }

    @Bean
    public Step parser(final Step parserWorker, final Partitioner partitioner) {
        return stepBuilderFactory.get("parser")
                .partitioner("parser", partitioner)
                .step(parserWorker)
                .taskExecutor(taskExecutor())
                .build();
    }

    @Bean
    public Step parserWorker(
            final StaxEventItemReader reader,
            final FlatFileItemWriter<Employee> writer) {
        return stepBuilderFactory.get("parserWorker")
                .<Employee, Employee>chunk(Integer.parseInt(chunkSize))
                .reader(reader)
                .writer(writer)
                .build();
    }

    @Bean
    @StepScope
    public StaxEventItemReader<Employee> reader(final @Value("file:#{stepExecutionContext[file]}") Resource resource) {
        StaxEventItemReader<Employee> staxEventItemReader = new StaxEventItemReader<>();
        staxEventItemReader.setResource(resource);
        staxEventItemReader.setFragmentRootElementName("Employee");
        Jaxb2Marshaller unMarshaller = new Jaxb2Marshaller();
        unMarshaller.setClassesToBeBound(Employee.class);
        staxEventItemReader.setUnmarshaller(unMarshaller);
        return staxEventItemReader;
    }

    @Bean()
    public FlatFileItemWriter<Employee> fileWriter() {
        FlatFileItemWriter<Employee> fileWriter = new FlatFileItemWriter<>();
        fileWriter.setResource(new FileSystemResource("out.csv"));
        EmployeeAggregator lineAggregator = new EmployeeAggregator();
        fileWriter.setLineAggregator(lineAggregator);
        fileWriter.setLineSeparator(EMPTY_STRING);
        fileWriter.setHeaderCallback(new HeaderCallback());
        fileWriter.setFooterCallback(new FooterCallback());
        return innlesFileWriter;
    }

I get error org.springframework.batch.item.ItemStreamException: Output file was not created:

Upvotes: 1

Views: 2359

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31590

I have spring batch configuration which reads from multiple files and write mutiple file.

You can create an additional step that merges the output files. Since the output file is a flat file, this can be done without any issue (that would be a bit more problematic if the output file was an XML file since you need to deal with XML declaration, headers, etc when merging files).

Another technique is to use a staging area (a table, a queue, etc) and add a step that reads from the staging area and write to the final file.

Upvotes: 1

Related Questions