CODEWITHSUNDEEP
apache-camelactivemq-classic
David
David

Reputation: 579

Apache Camel splitting large file

I have a Camel route that needs to split a large file (600k lines of id's) into 600k individual messages and then push them onto an Activemq queue. How do I optimize the route from the Camel side to increase throughput? I'm currently achieving ~150 messages/second throughput to AMQ. Here's what the route looks like currently. Any suggestions are appreciated!

from("file://directory")
  .split().jsonpath("$.ids").streaming().parallelProcessing()
  .log(LoggingLevel.INFO, "Split: ${body}")
  .to("activemq:queue:myqueue");

Upvotes: 3

Views: 675

Answers (1)

Petter Nordlander
Petter Nordlander

Reputation: 22279

First things first, as pointed out by @Bedla, Pool your connection (i.e. wrap your connection factory inside a org.apache.activemq.pool.PooledConnectionFactory).! It will most likely give you a throughput boost in the range of x10 to x100 depending on network conditions, message size etc. More for smaller messages.

Then, when hunting throughput, dumping each of the 600k lines into your log file will do no one no good. Remove it or at least put it to trace/debug level.

If your broker is located elsewhere, like other part of the world, or a place with bad network latency in general, consider using async dispatch on the ConnectionFactory setting. It will not wait for the roundtrip of a confirmation of each message sent.

Then finally, if none of the above give decent result (I think just a pool should do) turn off message persistence. The broker disk might be a bottle neck for low spec/old servers. There are even tweaks to enhance certain OS/storage combos for performance.

Upvotes: 1

Related Questions