Reputation: 4330

Celery: auto discovery does not find tasks module in app

I have the following setup with a fresh installed celery and django 1.4:

settings.py:

import djcelery
djcelery.setup_loader()

BROKER_HOST = 'localhost'
BROKER_PORT = 5672
BROKER_USER = 'user'
BROKER_PASSWORD = 'password'
BROKER_VHOST = 'test'

[...]

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.admin',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.staticfiles',
    'djcelery',
    'south',
    'compressor',
    'testapp',
]

testapp/tasks.py:

from celery.task import task

@task()
def add(x, y):
    return x + y

Message delivery to the celeryd works fine, but the task is always unregistered (so auto discovery does not seem to work correctly here). Only if I import the tasks module in tasks/__init__.py the task is found and I can use it.

Also the documentation was a little confusing about the decorator import, but I think this is the right one now.

Where is the bug in my setup?

Upvotes: 16

Answers (7)

Santos Solorzano

Reputation: 96

Adding on to Umair A.'s answer, there was an ImportError exception in my tasks.py file that was causing Celery to not register the tasks in the module. All other module tasks were registered correctly.

This error wasn't evident until I tried importing the Celery task within a Python Shell. I fixed the bad import statement and then the tasks were successfully registered.

Upvotes: 0

Umair A.

Reputation: 6873

In my case, I couldn't figure out the problem until I tried to import the tasks in shell (python or python manage.py shell).

>>> from project_name.tasks import task_add

Upvotes: 3

Janusz Skonieczny

Reputation: 19030

For any one who stumbles here looking for similar problem solution.

In my case it was switching from old module bases INSTALLED_APPS setting to a new AppConfig based configuration.

New applications should avoid default_app_config. Instead they should require the dotted path to the appropriate AppConfig subclass to be configured explicitly in INSTALLED_APPS.

To fix this you should change the way you feed packages to celery, as stated here in the 2248 Celery issue:

from django.apps import apps
app.autodiscover_tasks(lambda: [n.name for n in apps.get_app_configs()]

Instead of the old Celery 3 way:

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Upvotes: 11

dgel

Reputation: 16806

Add CELERY_IMPORTS to your settings.py:

CELERY_IMPORTS = ('testapp.tasks',)

Import all the tasks in testapp.tasks.__init__ file

Then Celery will import all tasks from testapp.tasks folder and name them as they are

Upvotes: 40

Iman Akbari

Reputation: 2204

I'm pretty sure you have to import the "Celery app instance" and declare a task like this:

from project_name.celery import app

@app.task
def video_process_task(video_id):
    pass

note that there's a celery.py file in the project_dir/project_name folder, which declares the celery instance, like this:

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project_name.settings')
app = Celery('project_name')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

as instructed in the official documentation.

also, you must register the task, in one of these three ways:

the CELERY_IMPORTS variable in Django project's settings.py file as in dgel's answer
passing bind=True to the decorator like: @app.task(bind=True)
if you set autodiscovery for celery like it is done above in the line app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) , each Django application's folder can contain a tasks.py file, and the tasks declared inside it will be automatically registered

note that changes to the tasks REQUIRE A CELERY RESTART to take effect

Upvotes: 2