How to add custom model to django_celery

Question

I'm working on making celery suitable for high availability I've forked the django_celery project and this fork of celery in order to make the customizations that I need. The celery link shows the modifications to the beat.py that utilizes the following code:

I've added this Lock model to django_celery models.py file and was able to migrate just fine:

from django.db import models

@python_2_unicode_compatible
class Lock(models.Model):
    name = models.CharField(max_length=127, unique=True)
    created = models.DateTimeField(auto_now=True)

    class Meta:
        verbose_name_plural = _('locks')

    def __str__(self):
        return self.name

In celery in the utils folder I've added this locked.py file:

from djcelery.models import Lock
from datetime import datetime, timedelta
from django.db import transaction, IntegrityError


class Locked(object):
    """A context manager to add a distributed mutex."""

    def __init__(self, name, timeout):
        self.name = name
        self.lock = None
        self.timeout = timeout

    def __enter__(self):
        # first delete any expired locks
        expired = datetime.utcnow() - timedelta(seconds=self.timeout)
        Lock.objects.filter(name=self.name, created__lte=expired).delete()
        # then try to get the lock
        try:
            Lock(name=self.name).save()
        except IntegrityError:
            transaction.rollback()
            raise LockError('Could not acquire lock: {0}'.format(self.name))

    def __exit__(self, *args):
        Lock.objects.filter(name=self.name).delete()


class LockError(Exception):
    """Exception thrown when the requested lock already exists."""

    pass

with these changes I'm able to run the following commands:

celery worker
python manage.py runserver
python manage.py shell

I issues arrises when I try to run the scheduler:

celery beat

I get the following error:

Traceback (most recent call last):
  File "venv/bin/celery", line 11, in <module>
    load_entry_point('celery', 'console_scripts', 'celery')()
  File "/venv/src/celery/celery/__main__.py", line 30, in main
    main()
  File "/venv/src/celery/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/venv/src/celery/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/venv/src/celery/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/venv/src/celery/celery/bin/celery.py", line 785, in handle_argv
    return self.execute(command, argv)
  File "/venv/src/celery/celery/bin/celery.py", line 717, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/venv/src/celery/celery/bin/base.py", line 315, in run_from_argv
    sys.argv if argv is None else argv, command)
  File "/venv/src/celery/celery/bin/base.py", line 377, in handle_argv
    return self(*args, **options)
  File "/venv/src/celery/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/venv/src/celery/celery/bin/beat.py", line 72, in run
    beat = partial(self.app.Beat,
  File "/venv/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/venv/src/celery/celery/app/base.py", line 572, in Beat
    return self.subclass_with_self('celery.apps.beat:Beat')
  File "/venv/src/celery/celery/app/base.py", line 504, in subclass_with_self
    Class = symbol_by_name(Class)
  File "/venv/lib/python2.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/venv/src/celery/celery/apps/beat.py", line 19, in <module>
    from celery import VERSION_BANNER, platforms, beat
  File "/venv/src/celery/celery/beat.py", line 35, in <module>
    from .utils.locked import Locked, LockError
  File "/venv/src/celery/celery/utils/locked.py", line 1, in <module>
    from djcelery.models import Lock
  File "/venv/src/django-celery/djcelery/models.py", line 30, in <module>
    class TaskMeta(models.Model):
  File "/venv/lib/python2.7/site-packages/django/db/models/base.py", line 105, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/venv/lib/python2.7/site-packages/django/apps/registry.py", line 237, in get_containing_app_config
    self.check_apps_ready()
  File "/venv/lib/python2.7/site-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

I have djcelery in my INSTALLED_APPS setting so I don't know what is going on at this point?

Answer 1

You must specify app instance to use for the celery command

-A APP, --app=APP app instance to use (e.g. module.attr_name)

For example if i have structure

pybilling
- pybilling
  - celeryconfig.py

then i should start celery beat with the command

celery --app pybilling.celeryconfig:app beat

Here is the contents of celeryconfig.py

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'pybilling.settings')

from django.conf import settings  # noqa

app = Celery('pybilling')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Answer 2

Celery is often used with Django and is compatible with Django but is is not inherently a Django application. The modifications you've made are such that when you run celery beat, Django models are loaded. In order to be able to use the models, the applications must be initialized first. The standard way to do this is to call django.setup() after having set things so that Django's code can find the Django settings. It could be something like:

import os
import django

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
django.setup()

You need to change project.settings to the actual module name that contains the settings.