CODEWITHSUNDEEP
phprabbitmq
williamvicary
williamvicary

Reputation: 805

Simultaneously creating messages on different queues

I have a script which picks up messages from a queue, this does the pre-processing required for the other processes to work.

Now, these messages have to be delivered so I need to ack these messages and if one of the services listening for messages goes down then it should receive the messages it missed when it comes back online.

A couple questions:

1/ Does it make sense to have a queue for each post-processing service which is added to everytime the pre-processing runs? (So I might add to 8 different queues at the same time following each process - this will be a ton of messages (hundreds of thousands p/day).

2/ How quick is it to add messages to a queue? Is adding to 8-10 queues going to slow down my software?

3/ Can I use a topic exchange to do this with fanout? My only concern is if one of my services goes down they will miss the message.

4/ Any tips from persons with experience?

Upvotes: 0

Views: 100

Answers (1)

kzhen
kzhen

Reputation: 3128

A few thoughts:

If your post-processors are each doing a different 'job' then it makes sense to have queues for them to consume from. If you just have a bunch of post-processors all doing the same task, then you only need to have one queue from which they can all consume messages from.

Adding messages to queues is FAST, adding queues into RabbitMQ is fast, binding the queues to exchanges is fast. The thing that will slow down your system would be the size of the messages and the number that you are likely to receive, and then how much processing actually needs to be done.

The other consideration is to do with persistence of messages, should your messages survive a restart of RabbitMQ, that is, how critical are they? if it is critical that they not be lost (which by the sounds of your question it is) then you will need to make sure they are persisted. If you look at the RabbitMQ documentation you will see that there is a significant cost in doing this.

This depends on what your system is actually doing...Topcis are good, Fanouts are good, but what your system does depends on which is applicable.

I would highly recommend reading RabbitMQ in Action it is an excellent resource and well worth the money.

Upvotes: 1

Related Questions