CODEWITHSUNDEEP
pythonrabbitmqcelery
Hick
Hick

Reputation: 36404

How to use celery to get messages from a rabbit-mq server?

The celery docs suggest that Rabbit-MQ must act like a middleman, where it is used as a messaging platform.

In my infrastructure, Rabbit-MQ is the primary server that serves me with some data every second. Now, whenever the data is served, I want Celery to do certain tasks.

Now, this throws out the whole publisher-worker model, as we're not using Celery where the messages are being produced.

So, how do I go about this?

Upvotes: 6

Views: 2719

Answers (3)

DejanLekic
DejanLekic

Reputation: 19787

Celery is not made to be arbitrary MQ consumer. If you want that, you should look into Kombu - messaging library for Python (which is also a Celery sub-project).

What you have to do is to write a tiny service based on Kombu that would consume arbitrary messages from RabbitMQ, and send Celery tasks to be executed on your Celery cluster. You could use some other library to communicate with RabbitMQ, but since you want to ultimately execute Celery tasks you will have Celery installed anyway, which in turn will install one of its main dependencies - Kombu. To communicate with RabbitMQ Kombu uses either py-amqp or (if installed) librabbitmq.

Upvotes: 2

skybunk
skybunk

Reputation: 863

A custom consumer seems like the best way to integrate this with Celery.

Have a look at http://docs.celeryproject.org/en/latest/userguide/extending.html#id4

Upvotes: 1

istepaniuk
istepaniuk

Reputation: 4271

Celery can use several back-ends. If you are already using RabbitMQ, it makes that option attractive.

These are however different concerns. Use a generic RabbitMQ client library such as pika to implement a consumer for your messages, then, if needed, use Celery to schedule tasks.

Upvotes: 2

Related Questions