CODEWITHSUNDEEP
javamultithreadingconcurrencyapache-camelthreadpool
Mike Barlotta
Mike Barlotta

Reputation: 725

Camel Split EIP and use of threads in pool doesn't seem to go over min threads

I am using Apache Camel 2.15 and found an interesting behavior. I am placing data received through a REST API call into a Camel route that is a direct endpoint. This route in turn uses the split EIP and calls another Camel route that is also a direct endpoint.

Here is what the relevant Camel code looks like

from("direct:activationInputChannel").routeId("cbr_activation_route")
// removed some processes
.split(simple("${body}"))
   .parallelProcessing()
   .to("direct:activationItemEndPoint")
.end()
// removed some processes

and 

from("direct:activationItemEndPoint").routeId("cbr_activation_item_route")
.process(exchange -> this.doSomething(exchange))
// removed some processes

The use of the direct endpoint should cause the calls to be synchronous and run on the same thread. As expected the split/parallelProcessing usage causes the second route to run on a separate thread. The splitter is using the default thread pool.

When I ran some load tests against the application using jMeter I found that the split route was becoming a bottleneck. With a load test using 200 threads I observed the Tomcat http thread pool as having a currentThreadCount of 200 in jConsole. I also observed the Camel route cbr_activation_route had 200 ExchangesInflight.

enter image description here

The problem was the cbr_activation_item_route only had 50 ExchangesInflight. The number 50 corresponded to the poolSize set for the default pool. The maxPoolSize was set to 500 and the maxQueueSize was set to 1000 (the default).

The number of inflight exchanges for this route never rose above the min pool size. Even though there were plenty of requests queued up and threads available. When I changed the poolSize in the Camel default thread pool to 200 then the cbr_activation_item_route used the new min value and had 200 ExchangesInflight. It seems that Camel would not use more threads than the minimum even when there were more threads available and even when under load.

Is there a setting or something that I could be missing that is causing this behavior? Why wouldn't Camel use 200 threads in the first test run when the minimum was set to 50?

Thanks

Upvotes: 4

Views: 2302

Answers (2)

codeperfector
codeperfector

Reputation: 31

Agree with Frederico's answer about the behavior of Java's Thread Pool Executor. It prefers to add new requests to the queue instead of creating more threads after 'corePoolSize' threads have been reached.

If you want your TPE to add more threads as requests come in after 'corePoolSize' has been reached, there is a slightly hacky way of achieving this based on the fact that Java calls the offer() method in BlockingQueue to queue the requests. If the offer() method returns false, it creates a new thread and calls the Executor's rejectedExecutionHandler. It is possible to override the offer() method and create your own version of the ThreadPool executor that can scale the number of threads based on load.

I found an example of this here: https://github.com/kimchy/kimchy.github.com/blob/master/_posts/2008-11-23-juc-executorservice-gotcha.textile

Upvotes: 3

fps
fps

Reputation: 34470

That's the expected behavior. This has nothing to do with Camel itself, but with Java's ThreadPoolExecutor in general.

If you read the linked docs, there it says:

If there are more than corePoolSize but less than maximumPoolSize threads running, a new thread will be created only if the queue is full.

If you set maxQueueSize to 1000, you have to create 1050 requests before new threads are created, up to 200. Try telling Camel to use a SynchronousQueue if you don't want your requests to be queued (not recommended, IMHO).

Upvotes: 1

Related Questions