CODEWITHSUNDEEP
djangodjango-celerykombu
Tim Richardson
Tim Richardson

Reputation: 7251

Django, celery and kombu: how to allow pickle?

I want to use picker as the celery serializer. I'm getting a kombu error that pickle is not allowed.

kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)

The exception occurs when I try to submit a task with .delay This task has a datetime argument.

Celery documentation says (https://docs.celeryproject.org/projects/kombu/en/master/userguide/serialization.html#guide-serialization)

By default Kombu will only load JSON messages, so if you want to use other serialization format you must explicitly enable them in your consumer by using the accept argument:

But I have no idea how to implement that.

I have

celery[redis]==5.1.2

and in my project's settings.py I have tried

CELERY_TASK_SERIALIZER = 'pickle'

which leads to the error.

(This setting is not documented here, is it old? https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-settings)

This is the content of celery.py I tried to understand Celery not accepting pickle even after allowing it

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from kombu import serialization

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.

app.config_from_object('django.conf:settings', namespace='CELERY')

accept_content = ['pickle', 'application/x-python-serialize']
task_serializer = 'pickle'
result_serializer = 'pickle'
serialization.register_pickle()
serialization.enable_insecure_serializers()

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

Upvotes: 2

Views: 2293

Answers (1)

Tim Richardson
Tim Richardson

Reputation: 7251

So this works, it seems:

in settings.py

CELERY_BROKER_URL = 'redis://redis:6379/0'
CELERY_RESULT_BACKEND = 'redis://redis:6379/0'
CELERY_TASK_SERIALIZER = 'pickle'   # changed, was json
CELERY_ACCEPT_CONTENT = ['json', 'pickle']  # new
os.environ.setdefault('C_FORCE_ROOT', 'true')  # new

there are many warnings about the evils of what I'm doing. This is running in docker on a single-tenant machine.

I did this because it was annoyed about a datetime parameter not working somehow. I don't understand why the solution is so complex. Django has a json serializer that handles datetime. Why can't celery use it?

Upvotes: 2

Related Questions