CODEWITHSUNDEEP
pythondjangorabbitmq
Sile
Sile

Reputation: 113

Usage of RabbitMQ queues with Django

I'm trying to add some real-time features to my Django applications, for that i'm using RabbitMQ and Celery on my django project, so what i would like to do is this: i have an external Python script which sends data to RabbitMQ > from RabbitMQ it should be retrieved from the Django app.

I'm sending some muppet data, like this:

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='Test')

channel.basic_publish(exchange='',
                      routing_key='Test',
                      body='Hello world!')

print(" [x] Sent 'Hello World!'")

connection.close()

What i would like to do is: as soon as i send Hello World!, my Django app should receive the string, so that i can perform some operations with it, such as saving it on my database, passing it to an HTML template or simply printing it to my console.

My actual problem is that i still have no idea how to do this. I added Celery to my Django project but i don't know how to connect to RabbitMQ and receive the message. Would i have to do it with Django Channels? Is there some tutorial on this? I found various material about using RabbitMQ and Celery with Django but nothing on this particular matter.

Upvotes: 0

Views: 4842

Answers (1)

Risadinha
Risadinha

Reputation: 16661

This is not directly connected to Celery.

You could solve it like this:

  1. create a manage command in Django that does whatever needs to be done with the incoming message/data (https://docs.djangoproject.com/en/3.0/howto/custom-management-commands/)
  2. create a consumer which could also be a Django command that is then started in a separate process, it is not part of the regular Django process (though it can be part of your Django code). In this consumer, you listen to the queue and whenever data comes in, the manage command from (1.) is called.

Of course, 1. and 2. could also be done in one single command. I've separated it to illustrate better the different aspects. And you might have different tasks and reuse one consumer. Also, if you have already (1.) you can reuse it like this, and you can test it easily without the overhead of the consumer.

More on RabbitMQ python consumers: https://github.com/celery/celery/blob/master/celery/worker/consumer/consumer.py

Here is the Celery consumer: https://github.com/celery/celery/blob/master/celery/worker/consumer/consumer.py because Celery of course has it's own consumer. It looks rather generic though, a simple consumer should be less complex.

(I only ever have written NSQ python consumers as part of Django, so no direct experience with RabbitMQ consumers (only as backend for Celery).)

EDIT: What you should ask yourself is - do I want the realtime data saved and stored in my Django app, first of all?

If yes - then RabbitMQ+Consumer is a very valid approach.

If no, if this is just for the user - you could also think about directly exposing it via an API to your frontend (and there use ajax calls to fetch it).

If no but you want to buffer the data to avoid hitting the other app that generates the data - then a queue is a very nice tool. In this case though, you might change the consumer not to save the data but to expose it to your frontend. If you only have to support new browsers you could use websockets which are supported now with Django 3:

https://blog.heroku.com/in_deep_with_django_channels_the_future_of_real_time_apps_in_django

Upvotes: 4

Related Questions