Prevent messages from going into the _error queue (masstransit, rabbitmq)

Question

I am successfully using MassTransit (with RabbitMQ) implementing a competing consumers topology: I can have 0 or many consumers on a queue, each of them prefetching and consuming exactly 1 message at a time (classic load balancing scenario).

For handling possible failures I can imagine 2 scenarios:

• the consumer crashed (without ACK on the message) => the message remains in queue if no other consumer is registered, or gets passed to the next running one
• the consumer throws an exception while processing data => the message goes to the _error queue -> can I avoid this? I would like that in this case the message follows the same rules as the step above: remains in queue if no other consumers or gets passed to a running one (even if it's the one which failed before).

Is this possible with MassTransit? Is this a bad strategy? Would it be better to gracefully handle processing inside the consumer and "never" throw?

Many thanks in advance!

Answer 1

When the consumer process crashes, the message isn't ACKed and stays in the queue, so there's nothing to do for your first scenario.

The error queue is the MassTransit implementation of the poison message queue. When a consumer throws, there are two options:

Transient failure

Transient failure when the consumer tries to reach other infrastructure, like a database (network down, database down, timeout, etc). Such failures are recoverable. MassTransit allows you to configure retry policies to handle specific exceptions by re-processing the message when the consumer has thrown an exception.

When the retry policy exceeds the number of configured retries or the exception is not handled by the policy, the failure is considered permanent.

Permanent failure

Some messages could be poisonous, so it doesn't matter how many times you retry, they won't be processed. For example, if you try to insert a record in a table with a primary key that is already there.

Such messages must be moved to the poison queue. Otherwise, the whole message processing will stop in an infinite retry loop. That's why those messages are called poison messages, and that's why the poison message queue exists. You can look at this queue to understand why the message got there, and if you need to fix the producer of this message, or the consumer to handle the message better.