CODEWITHSUNDEEP
apache-camel
Darius X.
Darius X.

Reputation: 2937

Shutdown Camel JMS consumer without losing incoming messages

We have OpenShift Pods using Camel to consumer from MQ (JmsComponent) When we redeploy, the PODs are shut down and new ones redeployed, with at least one Pod always consuming to the Queue.

We see Camel performing a graceful shutdown:

o.a.camel.impl.DefaultShutdownStrategy: Starting to graceful shutdown 17 routes (timeout 300 seconds)

But, if there are incoming messages on the Queue, we sometimes get this error:

DefaultJmsMessageListenerContainer: Rejecting received message because of the listener container having been stopped in the meantime:
  1. From the Camel Docs, it appears that we might need to set acceptMessagesWhileStopping=true. Is that correct? Any downsides?
  2. Do we need to implement a shutdown strategy that suspends the route first?

Upvotes: 2

Views: 1107

Answers (1)

burki
burki

Reputation: 7005

If you really lose messages (from the subject of your question) then you have to consume transacted. This is the only solution for message loss.

When you don't consume transacted, every message the broker sends to a consumer is done from the brokers perspective. So if the consumer loses it for whatever reason, it is lost.

When you consume transacted, the broker keeps the messages until the consumer commits the transaction. If the transaction is not committed, the broker sends the message(s) again (redelivery). If the configured number of redeliveries fails, the broker moves the message to a DLQ.

So, in your case of pod shutdown and recreation it is very important to consume transacted!

Imagine the case where some messages are sent to a pod but the pod is removed due to a redeployment. Without transactions, the messages are done because the pod has consumed it. That it wasn't able to finish processing of the messages doesn't matter.

In contrast, with transactions, some messages are sent to a pod, the pod goes down but the broker hasn't received a commit for the messages. The broker then sends the messages to another consumer (pod).

So in your deployment scenario it could happen (say hello to Murphy) that messages fail multiple times because they are delivered multiple times to pods that are just a moment later removed because of a redeployment. However, they are never lost. In the worst case, they are put aside in a DLQ.

When the JMS consumer is a Camel route, the transaction is normally committed when the route is successfully processed (this varies depending on the components used).

The setting acceptMessagesWhileStopping is then just "cosmetics". Because if incoming messages are immediately rejected when a shutdown is initiated, a lot of DLQ messages can pile up. Setting acceptMessagesWhileStopping=true can reduce this.

Upvotes: 3

Related Questions