CODEWITHSUNDEEP
pythondjangocelery
Alexander Tyapkov
Alexander Tyapkov

Reputation: 5067

Django + Celery + SQS setup. Celery connects to default RabbitMQ via ampq

I am trying to setup Amazon SQS as a default message broker for Celery in Django app. Celery worker is starting but broker is set to default RabbitMQ. Below you can find the output of my worker.

Here are some configs which I have in the project. My celery.py looks like:

from __future__ import absolute_import
import os
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'dance.settings')

app = Celery('dance')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

The essential part of the Django celery settings responsible for setup of broker url is:

BROKER_URL = 'sqs://{}:{}@'.format(AWS_ACCESS_KEY_ID, quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT_OPTIONS = {
    'region': 'eu-west-1',
    'polling_interval': 3,
    'visibility_timeout': 300,
    'queue_name_prefix':'dev-celery-',
}

When I am trying to launch worker within virtual environment with:

celery -A dance worker -l info

I receive following output:

 -------------- celery@universe v4.0.0 (latentcall)
---- **** ----- 
--- * ***  * -- Linux-4.8.0-28-generic-x86_64-with-debian-stretch-sid 2016-12-02 14:20:40
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         dance:0x7fdc592ca9e8
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


    [tasks]
...
    task1
    task2
...

Tasks are listed so I guess Celery gets and processes related Django settings. If to switch in settings from SQS to Redis, I get the same problem.

As I understand from read tutorials worker's output should look similar to.

- ** ---------- .> transport:   sqs://*redacted*:**@localhost//
- ** ---------- .> results:     djcelery.backends.database:DatabaseBackend

Also I am not using djcelery as far as it is outdated. Instead I am using django_celery_results as it is recommended on Celery setup pages. The last output is just a guess from side project.

Upvotes: 2

Views: 3520

Answers (2)

Andrii
Andrii

Reputation: 770

Incorrect: you need to use CELERY_BROKER_URL when you use CELERY namespace. But some options by default go with CELERY prefix, for example, CELERY_RESULT_BACKEND. If you use CELERY namespace so you need to write CELERY_CELERY_RESULT_BACKEND.

Upvotes: 2

Alexander Tyapkov
Alexander Tyapkov

Reputation: 5067

The only possible solution which I have found is to explicitly specify broker and database backend.

For me it looks strange, because settings from Django settings.py are not fully loaded or probably I have missed something, otherwise it is bug of Celery.

app = Celery('dance', broker='sqs://', backend='django-db')

Real solution:

Here is why I had problems:

All the Celery variables in Django should start with CELERY so instead of using BROKER_URL and BROKER_TRANSPORT_OPTIONS I had to use CELERY_BROKER_URL and CELERY_BROKER_TRANSPORT_OPTIONS

Upvotes: 4

Related Questions