CODEWITHSUNDEEP
javaspringspring-batch
membersound
membersound

Reputation: 86925

How to start the same spring-batch job concurrently with different parameters?

I want to start the same job configuration multiple times with different filename parameters, to import multiple files concurrently (each job one file).

@Configuration
public class MyJob {

    //the steps contain each a reader, processor and writer
    //reader is @JobScope
    @Bean
    public Job job(Step someStep, Step, someMoreStep) {
        System.out.println("registering job bean");
        return jobBuilderFactory.get(name)
            .start(someStep)
            .next(someMoreStep)
            .build();
    }
}

@Component
public class MyImporter {
    @Autowired
    private JobRegistry jobRegistry;

    @Autowired
    private JobLauncher launcher;

    private static final String FILENAME = "baseFilename";

    @Async
    public void run(String i) {
        p = new JobParametersBuilder();
        p.addDate("date", new Date());
        p.addString("filename", FILENAME + i + ".csv"); //injected via @Value jobParameter to job

        Job job = jobRegistry.getJob("getMyJob");
        launcher.run(job, p.toJobParameters());
    }
}

@Component
public class MyImportManager {
    @Autowired
    private MyImporter importer;

    //starts a  job multiple times with different "filename" parameters,
    //to simulate concurrent file imports with the same configuration job class
    public void start() {
        for (int i = 0; i < 4; i++) {
            importer.run(i);
        }
    }
}

@SpringBootApplication
@EnableBatchProcessing(modular = true)
@EnableAsync
public class MyConfig {

}

Problem: I can start multiple jobs, but they seem to use eg the same reader. If I run a single job i < 1, everything is working fine. If I increase i, I'm getting weired inputs.

I'm using FlatFileItemReader where each value is read line by line. But when using concurrent imports (from different files), the input strings are often corrupt.

So I assume I'm not registering the jobs properly for concurrent imports? But why?

What is interesting: the line "registering job bean" is only printed once. But shouldn't it be printed as often as the async job is started?

Upvotes: 1

Views: 1704

Answers (1)

Dean Clark
Dean Clark

Reputation: 3878

FlatFileItemReader is not thread-safe, so your reader bean will need to be in "job" scope. Then you'll get one instance per job.

@Bean
@JobScope
public FlatFileItemReader<?> yourReaderBean(
        @Value("#{jobParameters[filename]}") String filename){
    FlatFileItemReader<?> itemReader = new FlatFileItemReader<?>();
    itemReader.setLineMapper(lineMapper());
    itemReader.setResource(new ClassPathResource(filename));
    return itemReader;
}

Upvotes: 2

Related Questions