CODEWITHSUNDEEP
spring-integration
bbrower
bbrower

Reputation: 43

Spring integration thread limitations

We've got an SI application we use to route messages, do some validation, transformation, and filtering.

Essentially it consumes JMS messages from various queues and routes them to preregistered downstream outbound JMS queues.

We manage all of these through spring configs.

Inside of 1 JVM we have a few hundred(400-500) queue channels and associated pollers.

We're noticing as we scale this up (adding new inbound + outbound routes) we're having some decreased performance.

We don't seem to be having an impact to the box (CPU, memory, etc.) all seem pretty under utilized.

When profiled the application looks like so,

We've load tested the application from a volume perspective, but are in the process of looking at it from a scale up of in inbound + outbound processing flows.

We're running several of these on the box (all pointing at different external JMS resources), yet we only see slowdowns on one of these JVMs(the one with the most inbound + outbound flows configured).

From a long term perspective we're trying to understand

  1. Does SI have a scaling limit when it comes to pollers + queue channels?
  2. Some limit probably, what symptoms, how to detect?
  3. What's the constraint here, thread context switching? something else?
  4. We're on an old version of SI (1) Have improvements been made to handle this type of architecture, or is a redesign needed?
  5. Alternative architectures for high concurrency, event style routing between multiple inbound and outbound channels? Each inbound and outbound channels still have a set of stuff to possibly run through(transformation, filter, validation)

Upvotes: 2

Views: 1095

Answers (1)

Gary Russell
Gary Russell

Reputation: 174739

Why so many queue channels? It is unusual, and usually unnecessary, to have more than one async handoff (if any) in a flow. It is often not necessary at all, especially when using message-driven inbound endpoints (such as the JMS message-driven adapter).

This is because the threading and concurrency can be entirely managed by the adapter's message listener container.

The most efficient configuration is to set the concurrency (number of threads) on the inbound adapter to that required and use direct channels throughout.

In fact this is required if you wish no possibility of lost messages; as soon as a message is handed off in a queue channel, the JMS message is ack'd (by default).

Upvotes: 2

Related Questions