CODEWITHSUNDEEP
pythonrediscelery
stoneage
stoneage

Reputation: 510

Python celery - tuples in arguments converted to lists

I noticed this when using the delay() function to asynchronously send tasks. If I queue a task such as task.delay(("tuple",)), celery will store the argument as ["tuple"] and later the function will get the list back and not the tuple. Guessing this is because the data is being stored into json.

This is fine for tuples, however I'm using namedtuples which can no longer be referenced properly once converted to a list. I see the obvious solution of switching the namedtuples out with dicts. Is there any other method? I couldn't seem to find anything in the configuration for celery.

I'm using redis as the broker.

Upvotes: 4

Views: 1528

Answers (2)

tomerh
tomerh

Reputation: 1

Celery uses Kombo for message serialization. Currently the default serializer is 'json' which doesn't help much when we want to deserialize NamedTuple. You can override the default serializer with your own one or just use 'simplejson' which, by default uses the option namedtuple_as_object=True (see simplejson).

The following solution worked for me:

from kombu import serialization

serialization.register('simplejson',simplejson.dumps, simplejson.loads, 'application/json', 'utf-8')

app = Celery(...) # define your app

app.conf.task_serializer = 'simplejson'

And that's it. From now on every task that imports app and uses the annotation

@app.task
def task(...)

will use simplejson serializer instead of json.

Upvotes: 0

The Real Bill
The Real Bill

Reputation: 15773

If you need to preserve the python native data structure I'd recommend using one of the serialization modules such a cPickle which will preserve the data structure but won't be readable outside of Python.

Upvotes: 1

Related Questions