CODEWITHSUNDEEP
djangocelery
tsantor
tsantor

Reputation: 544

Celery 4.1.0 and Django 2.0.2 app tasks not discovered

I have found this question and tried all recommendations as well as the accepted answer. Still no luck.

Here is my mysite/taskapp/celery.py:

import os
from celery import Celery
from django.apps import AppConfig
from django.conf import settings

if not settings.configured:
    # set the default Django settings module for the 'celery' program.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')  # pragma: no cover


app = Celery('mysite')


class CeleryConfig(AppConfig):
    name = 'mysite.taskapp'
    verbose_name = 'Celery Config'

    def ready(self):
        app.config_from_object('django.conf:settings', namespace='CELERY')
        app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))  # pragma: no cover

Here is my mysite's __init__.py:

from mysite.taskapp.celery import app as celery_app

__all__ = ['celery_app']

Here is an example mysite/myapp/tasks.py:

from celery import shared_task

@shared_task
def mytask():
    print('Do something')

When running the celery worker with celery -A mysite worker -l info here is the output:

celery@tim-office v4.1.0 (latentcall)

Darwin-16.7.0-x86_64-i386-64bit 2018-02-27 21:38:55

[config]
.> app:         mysite:0x1058e0fd0
.> transport:   amqp://guest:**@localhost:5672//
.> results:     disabled://
.> concurrency: 4 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.> celery           exchange=celery(direct) key=celery


[tasks]
. mysite.taskapp.celery.debug_task

Note that the mysite/myapp/tasks.py tasks are not found. I've been spinning my wheels for a full day trying all kinds of things from adding the app to CELERY_IMPORTS in the settings file:

CELERY_IMPORTS = ('myapp.tasks',)

To trying to force them into autodiscover_tasksline in the abovemysite/taskapp/celery.py`:

app.autodiscover_tasks(['myapp',])

Where am I going wrong here? Any help is greatly appreciated.

UPDATE: Possibly worth noting I am using the project structure of cookiecutter-django.

Upvotes: 1

Views: 2415

Answers (3)

TheRightChoyce
TheRightChoyce

Reputation: 3084

I had the same/a similar issue when using Django 2 and Celery 4.1. It appears that the AppConfig messes up some of the auto-configuration with Celery. I found two solutions:

  1. Don't pass in settings.INSTALLED_APPS to the app.autodiscover_tasks function. This may mess something else up down the road, but haven't run into it yet
  2. Do pass it in, but include an array with just the names of the apps as pointed out here https://stackoverflow.com/a/39129453/502572 (looks you like you mentioned a similar approach as a comment)

Here's my <project_name>/<project_name>/celery.py file:

from __future__ import absolute_import, unicode_literals
import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_name>.settings')

app = Celery('<project_name>')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

if __name__ == '__main__':
    app.start()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

My __init__.py and <project_name>/<app_name>/tasks.py is setup the same as yours.

Upvotes: 1

himenon
himenon

Reputation: 219

I had the same problem, but since I solved it safely, I will respond. My environment is as follows.

  • Django 4.0
  • Celery 2.0

First of all, I will show you the sample code. (I am sorry, the language is Japanese.

Sample Code

The main points for making Django recognize tasks are as follows.

  1. In order for Server to recognize Task, app.autodiscover_tasks () is called only once when loading settings.py specified byDJANGO_SETTINGS_MODULE. It does not need to be called on other sub applications.

  2. When starting worker and scheduler, specify DJANGO_SETTINGS_MODULE.

    # Server
python3 manage.py runserver 0.0.0.0:8000

# Worker
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm worker

# Scheduler
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm beat --scheduler django_celery_beat.schedulers:DatabaseScheduler

With these two, Django side should recognize the task. Please see Sample Code for details.

References:

Upvotes: 1

bdoubleu
bdoubleu

Reputation: 6127

I would comment but don't have 50 reputation.

Do you have logging setup? If so, temporarily disable logging and try to start celery again.

It's possible that there's an uncaught error when Django starts. In my case celery didn't have the correct permissions to a log file.

Upvotes: 1

Related Questions