CODEWITHSUNDEEP
pythondjangocelerydjango-celerycelery-task
Nathaniel Tucker
Nathaniel Tucker

Reputation: 597

Why does celery return a KeyError when executing my task?

I keep getting this keyError. I'm sending strings and id (integers) to the task function, so I don't think it is serialization issue. Also it says the keyerror is on the path to the function itself, not the contents. Please help.

Tasks.py:

from celery.decorators import task
from notification import models as notification

@task(ignore_result=True)
def notify_match_creation(match, home_team, away_team, home_team_captain, away_team_captain):
    notification.send(User.objects.filter(profile__teams__pk__in=(home_team, away_team)),
                      "tournaments_new_match",
                      {'match': unicode(match),
                       'home_team_captain': home_team_captain,
                       'away_team_captain': away_team_captain,
                       })

Relevant settings:

CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "postgresql://user:pass@localhost/ahgl"
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
BROKER_VHOST = "/"

Celery output:

[Tasks]

  . apps.tournaments.tasks.notify_match_creation
  . tournaments.tasks.notify_match_creation
[2012-02-25 02:34:06,209: WARNING/MainProcess] celery@NATTOWER has started.
[2012-02-25 02:34:06,477: WARNING/PoolWorker-4] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,479: WARNING/PoolWorker-2] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,523: WARNING/PoolWorker-3] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:06,566: WARNING/PoolWorker-1] E:\Webdesign\ahgl\ENV\lib\site-packages\djcelery\loaders.py:84: UserWarn
ing: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-02-25 02:34:31,520: INFO/MainProcess] Got task from broker: apps.tournaments.tasks.notify_match_creation[4dbd6258-
5cee-49e9-8c8a-2d2105a2d52a]
[2012-02-25 02:34:31,569: ERROR/MainProcess] Task apps.tournaments.tasks.notify_match_creation[4dbd6258-5cee-49e9-8c8a-2
d2105a2d52a] raised exception: KeyError('apps.tournaments.tasks.notify_match_creation',)
Traceback (most recent call last):
  File "E:\Webdesign\ahgl\ENV\lib\site-packages\celery\concurrency\processes\pool.py", line 211, in worker
    result = (True, func(*args, **kwds))
  File "E:\Webdesign\ahgl\ENV\lib\site-packages\celery\worker\job.py", line 50, in execute_and_trace
    task = tasks[name]
KeyError: 'apps.tournaments.tasks.notify_match_creation'
[2012-02-25 02:38:29,773: WARNING/MainProcess] celeryd: Hitting Ctrl+C again will terminate all running tasks!
[2012-02-25 02:38:29,773: WARNING/MainProcess] celeryd: Warm shutdown (MainProcess)
[2012-02-25 02:38:31,779: INFO/MainProcess] process shutting down

Upvotes: 15

Views: 33533

Answers (4)

Super Kai - Kazuya Ito
Super Kai - Kazuya Ito

Reputation: 1

I got the similar KeyError below for celery worker:

Traceback (most recent call last):
  File "C:\Users\kai\AppData\Local\Programs\Python\Python39\lib\site-packages\celery\worker\consumer\consumer.py", line 591, in on_task_received
    strategy = strategies[type_]
KeyError: 'account.tasks.display'

Because I have display task in store/tasks.py as shown below:

# "store/tasks.py"

from celery import shared_task

@shared_task
def display(arg):
    return arg

Then, I used the wrong path "account.tasks.display" to use display task in store/tasks.py as shown below:

# "core/settings.py"

CELERY_BEAT_SCHEDULE = {
    "scheduled_task": {
        "task": "account.tasks.display", # Here
        "schedule": 5.0,
        "args": ["Test"],
    }
}

So to solve the error, I used the correct path "store.tasks.display" as shown below, then the error was solved:

# "core/settings.py"

CELERY_BEAT_SCHEDULE = {
    "scheduled_task": {
        "task": "store.tasks.display", # Here
        "schedule": 5.0,
        "args": ["Test"],
    }
}

Upvotes: 0

Immanuel
Immanuel

Reputation: 67

Instead of using task.nameoftask it is rather a relative import as like

app.task.nameoftask

Upvotes: 0

Shankar ARUL
Shankar ARUL

Reputation: 13740

In the celery output, you see that the task that gets picked up is

tournaments.tasks.notify_match_creation (#1)

and in your key error it is

KeyError: 'apps.tournaments.tasks.notify_match_creation'

before you call the celery task, make sure that it is imported with the same name (structure) as it is in the celery task that gets picked up (#1). Please refer to the link from celery docs for getting your relative imports rights.

one possible solution, when you launch celery - try

celery worker -A apps.tournaments.tasks.notify_match_creation

this could align your task names

Upvotes: 8

Dr Manhattan
Dr Manhattan

Reputation: 14097

I had the exact same issue, however, the following worked for me,

in my tasks.py I changed from

from celery.task import task

@task(name="generate_and_email_membership_invoice")
def my_task_name():
      ...

to

from ..celeryconf import app # the file where you have app.autodiscover_tasks()

@app.task()
def my_task_name():
     ..

Upvotes: 0

Related Questions