CODEWITHSUNDEEP
javaparallel-processingapache-camel
Nagesh
Nagesh

Reputation: 347

Parallel processing of multiple files using Apache Camel

I have thousands of files to process each individually. I used Apache Camel to process the files

from("file:C:\datafiles\input?readLock=changed&readLockTimeout=100&readLockCheckInterval=20") .process(new MyProcessor()).to("file:C:\datafiles\output");

I do each file one at a time and it takes 30 minutes.

I'm thinking I can do this processing in 10 simultaneous threads, 10 files at a time, and I might be able to do it in 3 minutes instead of 30.

My question is, what is the "correct" way to achieve my 10 threads? And when one is done, create a new one to a max number of 10.

When I searched the internet I got some suggestions like using

  1. maxMesssagesPerPoll
  2. threads(10)
  3. Aggregators

But I do not have anything to do with Aggregators here, just need to process each file from one remote location, process it and then place them in another remote location.

Upvotes: 0

Views: 3669

Answers (1)

Nagesh
Nagesh

Reputation: 347

Can use the below code to achieve multithreading.

public class RouterConfig extends RouteBuilder {
    public void configure() throws Exception {
        from("file:C:\datafiles\input?readLock=changed&readLockTimeout=100&readLockCheckInterval=20&maxMessagesPerPoll=3") 
        .threads(3, 3, "myThread")
        .process(new MyProcessor())
        .to("file:C:\datafiles\output");
    }
}

Here threads api will create 3 threads. maxMessagesPerPoll=3 will pick 3 files at one poll and pass one file to one thread, like wise here it picks 3 files (file1, file2, file3) and gives one file 3 threads (thread1, thread2, thread3).

  • so, "file1" will be processed by "thread1" and
  • "file2" will be processed by "thread2"
  • "file3" will be processed by "thread3"

Upvotes: 3

Related Questions