CODEWITHSUNDEEP
pythonrabbitmqpika
m-ric
m-ric

Reputation: 5901

Limit number of connections to a rabbit queue?

I use pika-0.10.0 with rabbitmq-3.6.6 broker on ubuntu-16.04. I designed a Request/Reply service. There is a single Request queue where all clients push their requests. Each client creates a unique Reply queue: the server pushes replies targeting this client to this unique queue. My API can be seen as two messages: init and run.

init messages contain big images, thus init is a big and slow request. run messages are lighter and the server reuses previous images. The server can serve multiple clients. Usually client#1 init then run multiple times. If client#2 comes in and init, it will replace the images sent by client#1 on the server. And further run issued by client#1 would use wrong images. Then I am asking:

Upvotes: 0

Views: 719

Answers (1)

Alex Buyny
Alex Buyny

Reputation: 3185

I think you have a problem in your design. Logically each run corresponds to a certain init so they have to be connected. I'd put a correlation id field into init and run events. When server receives run it checks if it there was a corresponding init processed and uses the result of that init.

Speaking of performance: You can make init worker queue and have multiple processing servers listen to it. The example is in the RabbitMQ docs Then, when init request comes in, one of available servers will pick it up, and store your images and the correlation ID. If you have multiple init requests at the same time - no problem, they will be processed eventually (or simultaneosly if servers are free)

Then server that did the process sends reply message to the client queue saying init work is done, and sends name of the queue where run request have to be published.

When ready, client sends its run request to the correct queue.

To directly answer the question:

there is a common misconception that you publish to a queue. In RabbitMQ you publish to an exchange that cares about routing of your messages to a number of queues. So you question really becomes can I limit number of publishing connections to an exchange. I'm pretty sure there is no way of doing so on the broker side. Even if there was a way of limiting number of connections, imagine the situation:

  1. Client1 comes in, pushes its 'init' request.
  2. Client1 holds its connection, waiting to push run.
  3. Client1 fails or network partition occurs, its connection gets dropped.
  4. Client2 comes in and pushes its init request.
  5. Client2 fails
  6. Client1 comes back up and pushes its run and gets Client2's images.

Connection is a transient thing and cannot be relied upon as a transaction mechanism.

Upvotes: 1

Related Questions