CODEWITHSUNDEEP
javaspringmultithreadingspring-bootspring-batch
dominikbrandon
dominikbrandon

Reputation: 418

Spring Batch limits number of concurrent threads to 10

I've got Spring Boot application with Spring Batch framework. My goal is simple - run certain job concurrently. I want to be able to, let's say, run 15 threads in the same time and reject every excessive one. Here's my configuration class:

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.JobRegistry;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.launch.support.SimpleJobLauncher;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.TaskExecutor;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

@Configuration
@EnableBatchProcessing
public class GeneratingReportJobConfiguration {

    @Autowired
    private GeneratingReportTask task;

    @Autowired
    private JobRepository jobRepository;

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Bean
    public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
        JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
        jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
        return jobRegistryBeanPostProcessor;
    }

    @Bean
    public TaskExecutor taskExecutor() {
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(15);
        taskExecutor.setMaxPoolSize(15);
        taskExecutor.setQueueCapacity(0);
        return taskExecutor;
    }

    @Bean
    public JobLauncher jobLauncher(TaskExecutor taskExecutor) {
        SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
        jobLauncher.setJobRepository(jobRepository);
        jobLauncher.setTaskExecutor(taskExecutor);
        return jobLauncher;
    }

    @Bean
    public Job job() {
        return jobBuilderFactory.get("generatingReportJob")
                .start(step())
                .build();
    }

    @Bean
    public Step step() {
        return stepBuilderFactory.get("generatingReportTask")
                .tasklet(task)
                .build();
    }
}

The thing is, whatever I'm trying to do - application runs at most 10 jobs at the same time. I've tried experimenting with TaskExecutor - my first logical choice was SimpleAsyncTaskExecutor, which appeared to stash every excessive request and then run them in a random order which is not intended.

So later I've tried to manipulate that limit and, as you can see in the code, I've started using ThreadPoolTaskExecutor, which allowed me to set the limit to 5, for example, and it worked as expected - at most 5 threads could run at the same time and next ones were being rejected while these 5 were running. However, setting the limit to 15 resulted in behaviour similar to the previous one. Still, 11th request was being queued, whereas 16th was being rejected. This is almost my intended behaviour, however I need to be able to fully control thread execution.

Jobs are being invoked via @RestController using jobLauncher.run(). Every excessive request results in browser loading until being able to start execution. Seems like, program is being frozen somewhere inside jobLauncher.run() (when able to start job execution it just exits - that's how async run works).

Upvotes: 5

Views: 5495

Answers (1)

ACH
ACH

Reputation: 195

Bit late but try to check if you are using HikariCP which has a default 10 simultaneous DB connexions limit

Just faced similar issue and resolved by increasing spring.datasource.hikari.maximum-pool-size

See: Number of parallel threads processing is capped to 10 with Spring Batch

Upvotes: 2

Related Questions