CODEWITHSUNDEEP
spring-batch
EagerToLearn
EagerToLearn

Reputation: 89

Spring Batch : How to make chunk processing as multi threaded

I have a spring Batch Chunk based job with (Custom Reader, Custom Processor, Custom Writer Any thread should follow this order). If i start 10 threads they should run in parallel doing their own sequence of reading,processing,writing in order.

Thread 1-->In order: Read, process, write

Thread 2-->In order: Read, process, write...etc

How to implement this behavior. Currently i have code working in single thread model as below. (I think. Not sure). I want commit-interval in batch:chunk as '1' only (Because i need to read file line by line).

<!-- Reader bean for our simple Text file -->
<bean id="productReader" class="com.mycompany.batching.reader.productItemReader"
    scope="step">
    <property name="userId" value="#{jobParameters['userId']}" />
    <property name="dataId" value="#{jobParameters['dataId']}" />
</bean>


<!-- Processor for the data red from text file -->
<bean id="productProcessor" class="com.mycompany.batching.processor.productItemProcessor">
</bean>

<!-- Persisting data to database -->
<bean id="productWriter" class="com.mycompany.batching.writer.productItemWriter">
</bean>

<batch:job id="simpleProductImportJob" xmlns="http://www.springframework.org/schema/batch">
    <batch:step id="importFileStep">
        <batch:tasklet>
            <batch:chunk reader="productReader" processor="productProcessor"
                writer="productWriter" commit-interval="1" />
        </batch:tasklet>
    </batch:step>
</batch:job>

How to make this multi threaded.?

Upvotes: 0

Views: 1705

Answers (1)

Jay
Jay

Reputation: 429

Best way to achieve this is by Step Partitioning. Suppose your source is a File, 1. Split the file based on some logic 2. Partitioner will pick the files and assign each files to the slaves.

batch-context.xml 

<batch:step id="fileProcessMaster">
        <batch:partition partitioner="filePartioner" step="readAndInsertToDb">
            <batch:handler task-executor="taskExecutor" />
        </batch:partition>
    </batch:step>

FilePartitioner.java

public class FilePartitioner implements Partitioner{

@Override
public Map<String, ExecutionContext> partition(int gridSize) {
     Map<String, ExecutionContext> map = new HashMap<String, ExecutionContext>();
        int i = 0;
        File[] files = new File(inboundDirectory).listFiles((FileFilter) new PrefixFileFilter(*);
        for (File file : files) {
            //Pick only those files which are splitted
            if (file.getName().contains("Split")) {
                ExecutionContext context = new ExecutionContext();
                context.putString("file", file.getAbsolutePath());
                map.put("slave" + i, context);
                i++;
            }
        }
        return map;
}

You have make sure that the bean configuration for partitioner is step scoped.

Upvotes: 0

Related Questions