Making sure a message published on a topic exchange is received by at least one consumer

Question

TLDR; In the context of a topic exchange and queues created on the fly by the consumers, how to have a message redelivered / the producer notified when no consumer consumes the message?

I have the following components:

a main service, producing files. Each file has a certain category (e.g. pictures.profile, pictures.gallery)
a set of workers, consuming files and producing a textual output from them (e.g. the size of the file)

I currently have a single RabbitMQ topic exchange.

The producer sends messages to the exchange with routing_key = file_category.
Each consumer creates a queue and binds the exchange to this queue for a set of routing keys (e.g. pictures.* and videos.trending).
When a consumer has processed a file, it pushes the result in a processing_results queue.

Now - this works properly, but it still has a major issue. Currently, if the publisher sends a message with a routing key that no consumer is bound to, the message will be lost. This is because even if the queue created by the consumers is durable, it is destroyed as soon as the consumer disconnects since it is unique to this consumer.

Consumer code (python):

channel.exchange_declare(exchange=exchange_name, type='topic', durable=True)
result = channel.queue_declare(exclusive = True, durable=True)
queue_name = result.method.queue

topics = [ "pictures.*", "videos.trending" ]
for topic in topics:
    channel.queue_bind(exchange=exchange_name, queue=queue_name, routing_key=topic)

channel.basic_consume(my_handler, queue=queue_name)
channel.start_consuming()

Loosing a message in this condition is not acceptable in my use case.

Attempted solution

However, "loosing" a message becomes acceptable if the producer is notified that no consumer received the message (in this case it can just resend it later). I figured out the mandatory field could help, since the specification of AMQP states:

This flag tells the server how to react if the message cannot be routed to a queue. If this flag is set, the server will return an unroutable message with a Return method.

This is indeed working - in the producer, I am able to register a ReturnListener :

rabbitMq.confirmSelect();  

rabbitMq.addReturnListener( (int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) -> {
    log.info("A message was returned by the broker");
});
rabbitMq.basicPublish(exchangeName, "pictures.profile", true /* mandatory */, MessageProperties.PERSISTENT_TEXT_PLAIN, messageBytes);

This will as expected print A message was returned by the broker if a message is sent with a routing key no consumer is bound to.

Now, I also want to know when the message was correctly received by a consumer. So I tried registering a ConfirmListener as well:

rabbitMq.addConfirmListener(new ConfirmListener() {
    void handleAck(long deliveryTag, boolean multiple) throws IOException {
        log.info("ACK message {}, multiple = ", deliveryTag, multiple);
    }

    void handleNack(long deliveryTag, boolean multiple) throws IOException {
        log.info("NACK message {}, multiple = ", deliveryTag, multiple);
    }
});

The issue here is that the ACK is sent by the broker, not by the consumer itself. So when the producer sends a message with a routing key K:

If a consumer is bound to this routing key, the broker just sends an ACK
Otherwise, the broker sends a basic.return followed by a ACK

Cf the docs:

For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack. The same is true for negative acknowledgements (basic.nack).

So while my problem is theoretically solvable using this, it would make the logic of knowing if a message was correctly consumed very complicated (especially in the context of multi threading, persistence in a database, etc.):

send a message

on receive ACK:
    if no basic.return was received for this message
       the message was correctly consumed
    else
       the message wasn't correctly consumed

on receive basic.return
    the message wasn't correctly consumed

Possible other solutions

Have a queue for each file category, i.e. the queues pictures_profile, pictures_gallery, etc. Not good since it removes a lot of flexibility for the consumers
Have a "response timeout" logic in the producer. The producer sends a message. It expects an "answer" for this message in the processing_results queue. A solution would be to resend the message if it hasn't been answered to after X seconds. I don't like it though, it would create some additional tricky logic in the producer.
Produce the messages with a TTL of 0, and have the producer listen on a dead-letter exchange. This is the official suggested solution to replace the 'immediate' flag removed in RabbitMQ 3.0 (see paragraph Removal of "immediate" flag). According to the docs of the dead letter exchanges, a dead letter exchange can only be configured per-queue. So it wouldn't work here
[edit] A last solution I see is to have every consumer create a durable queue that isn't destroyed when he disconnects, and have it listen on it. Example: consumer1 creates queue-consumer-1 that is bound to the message of myExchange having a routing key abcd. The issue I foresee is that it implies to find an unique identifier for every consumer application instance (e.g. hostname of the machine it runs on).

I would love to have some inputs on that - thanks!

Related to:

RabbitMQ: persistent message with Topic exchange (not applicable here since queues are created "on the fly")
Make sure the broker holds messages until at least one consumer gets it
RabbitMQ Topic Exchange with persisted queue

[edit] Solution

I ended up implementing something that uses a basic.return, as mentioned earlier. It is actually not so tricky to implement, you just have to make sure that your method producing the messages and the method handling the basic returns are synchronized (or have a shared lock if not in the same class), otherwise you can end up with interleaved execution flows that will mess up your business logic.

Making sure a message published on a topic exchange is received by at least one consumer

Attempted solution

Possible other solutions

[edit] Solution

Answers (1)

Related Questions