CODEWITHSUNDEEP
rabbitmq
Karlen Kishmiryan
Karlen Kishmiryan

Reputation: 7522

How to manage consumers' sequential work using RabbitMQ?

I am using RabbitMQ to handle a lot of data. I have three types of consumers: downloader, extracter and importer. They must work sequentially in the order I mentioned above e.g. the downloader will download a bunch of files, pass to extracter and after extracting them the importer should import the info from the files into DB. My question is how can I implement this kind of functionality? Is this is a good approach to put the messages into extracter's queue from downloader (consumer)? Or there is another solution to this?

Upvotes: 1

Views: 805

Answers (1)

Donut
Donut

Reputation: 112875

This seems like a great case to use three separate queues. You'd have a download queue, an extraction queue, and an import queue.

The consumer of each queue would be responsible for putting a message on the next queue in the pipeline after it has finished its work.

So the workflow would look like this (queues prefixed with queue. to make them obvious):

??? -> queue.download
queue.download -> downloader -> download complete -> queue.extraction
queue.extraction -> extracter -> extraction complete -> queue.import
queue.import -> importer -> import complete -> ???

A couple of things to note:

  • You'll need something to put messages into the download queue to kick things off.
  • You can add other steps in the pipeline easily -- want to add a compression step (for example)? Have the downloader put messages into a compression queue, and then once that step is complete, put a message in the extraction queue.
  • You can scale horizontally at any stage of the pipeline with ease. Downloading takes too long? Spin up more instances of downloader that pull messages off the same work queue.

Hope this helps!

Upvotes: 1

Related Questions