ImportError being generated when trying to run django-celery worker process

Question

I'm trying to integrate django-celery into an existing site and I'm coming up against an error that I can't seem to get fixed.

For context, I went through the Django first steps and the test project was successful, ie everything worked as it should.

Now, in my existing project, I can't get the celery worker running from the command line:

manage.py celery worker --loglevel=info --settings=myproject.settings.dev_settings

When i run that I get the following stack trace and error:

Traceback (most recent call last):
  File "C:\sites\corecrm\manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 453, in execute_from_command_line
    utility.execute()
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Python27\lib\site-packages\djcelery\management\commands\celery.py", line 22, in run_from_argv
    ['%s %s' % (argv[0], argv[1])] + argv[2:],
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 901, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "C:\Python27\lib\site-packages\celery\bin\base.py", line 187, in execute_from_commandline
    return self.handle_argv(prog_name, argv[1:])
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 893, in handle_argv
    return self.execute(command, argv)
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 868, in execute
    return cls(app=self.app).run_from_argv(self.prog_name, argv)
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 148, in run_from_argv
    return self(*args, **options)
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 118, in __call__
    ret = self.run(*args, **kwargs)
  File "C:\Python27\lib\site-packages\celery\bin\celery.py", line 220, in run
    return self.target.run(*args, **kwargs)
  File "C:\Python27\lib\site-packages\celery\bin\celeryd.py", line 153, in run
    return self.app.Worker(**kwargs).run()
  File "C:\Python27\lib\site-packages\celery\apps\worker.py", line 162, in run
    self.app.loader.init_worker()
  File "C:\Python27\lib\site-packages\celery\loaders\base.py", line 130, in init_worker
    self.import_default_modules()
  File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 138, in import_default_modules
    self.autodiscover()
  File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 141, in autodiscover
    self.task_modules.update(mod.__name__ for mod in autodiscover() or ())
  File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 176, in autodiscover
    for app in settings.INSTALLED_APPS])
  File "C:\Python27\lib\site-packages\djcelery\loaders.py", line 195, in find_related_module
    return importlib.import_module('%s.%s' % (app, related_name))
  File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\sites\corecrm\people\tasks.py", line 15, in <module>
    from people.models import Customer, CustomerCsvFile, CustomerToTag, get_customer_from_csv_row
  File "C:\sites\corecrm\people\models.py", line 163, in <module>
    UserProfile._meta.get_field_by_name('username')[0]._max_length = 75
  File "C:\Python27\lib\site-packages\django\db\models\options.py", line 351, in get_field_by_name
    cache = self.init_name_map()
  File "C:\Python27\lib\site-packages\django\db\models\options.py", line 380, in init_name_map
    for f, model in self.get_all_related_m2m_objects_with_model():
  File "C:\Python27\lib\site-packages\django\db\models\options.py", line 469, in get_all_related_m2m_objects_with_model
    cache = self._fill_related_many_to_many_cache()
  File "C:\Python27\lib\site-packages\django\db\models\options.py", line 483, in _fill_related_many_to_many_cache
    for klass in get_models(only_installed=False):
  File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 197, in get_models
    self._populate()
  File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 75, in _populate
    self.load_app(app_name)
  File "C:\Python27\lib\site-packages\django\db\models\loading.py", line 96, in load_app
    models = import_module('.models', app_name)
  File "C:\Python27\lib\site-packages\django\utils\importlib.py", line 35, in import_module
    __import__(name)
  File "C:\sites\corecrm\booking\models.py", line 17, in <module>
    from people.models import Customer, UserProfile
ImportError: cannot import name Customer

To try and work out what the booking/models.py script sees in people I added the following at the start:

import people
print 'path: %s' % people.__path__
for item in dir(people):
    print item

and that gives me the following output:

path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__

however, when I run manage.py shell --settings=myproject.settings.dev_settings I get the following output:

path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
path: ['C:\\sites\\corecrm\\people']
__builtins__
__doc__
__file__
__name__
__package__
__path__
models

As you can see the models module is available at the end of the 2nd list for the shell command (and I've confirmed this is also the case for manage.py commands other than celery). How would I make sure that module is available at the same point when I run the celery command?

EDIT: I've now also set up this project on an Ubuntu VM and I'm getting the same error when I try to run the worker manage command. Any ideas? Anyone?

ANOTHER EDIT: I've pasted the code for booking/models.py and people/models.py at http://pastebin.com/fTVVBtB4

Answer 1

I'm pretty sure this line is your problem:

File "C:\sites\corecrm\people\models.py", line 163, in <module>
    UserProfile._meta.get_field_by_name('username')[0]._max_length = 75

While you're still busy importing from people.models, this line (in particular get_field_by_name) forces Django to evaluate the model and setup all relationships between that model and it's related models. This, in turn, forces an import of Customer in people.models, while you're still busy importing that exact model. This is what results in an ImportError.

For a working solution you'll need to post your models.py.

Why does this error only occur with celery? I can't say for sure without some more information, but my best guess is that Celery handles importing everything slightly different (Django probably doesn't import Customer, CustomerCsvFile, CustomerToTag and get_customer_from_csv_row all at once) and that this exposes the bug in your code.

EDIT/SOLUTION:
I would remove this line:

UserProfile._meta.get_field_by_name('username')[0]._max_length = 75

And move it to the instance level, into the __init__ method:

class UserProfile(AbstractUser):
    def __init__(self, *args, **kwargs):
        self._meta.get_field_by_name('username')[0]._max_length = 75
        super(UserProfile, self).__init__(*args, **kwargs)

If the cause of the issue is indeed what I think it is, this will fix the circular import while providing the same functionality. If the max_length functionality gets broken somehow (most likely because internally a max_length validator is added to CharField and _max_length is changed too late) I would instead override the complete username field in the init method:

class UserProfile(AbstractUser):
    def __init__(self, *args, **kwargs):
        super(UserProfile, self).__init__(*args, **kwargs)
        self._meta.get_field_by_name('username')[0] = models.CharField(max_length=75, etc.)