CODEWITHSUNDEEP
python-3.xdjangocelery
automate
automate

Reputation: 53

Django model object as parameter for celery task raises EncodeError - 'object of type someModelName is not JSON serializable'

Im working with a django project(im pretty new to django) and running into an issue passing a model object between my view and a celery task.

I am taking input from a form which contains several ModelChoiceField fields and using the selected object in a celery task. When I queue the task(from the post method in the view) using someTask.delay(x, y, z) where x, y and z are various objects from the form ModelChoiceFields I get the error object of type <someModelName> is not JSON serializable.

That said, if I create a simple test function and pass any of the same objects from the form into the function I get the expected behavior and the name of the object selected in the form is logged.

def test(object):
    logger.debug(object.name)

I have done some poking based on the above error and found django serializers which allows for a workaround by serializing the object using serializers.serialize('json', [template]), in the view before passing it to the celery task.

I can then access the object in the celery task by using template = json.loads(template)[0].get('fields') to access its required bits as a dictionary -- while this works, it does seem a bit inelegant and I wanted to see if there is something I am missing here.

Im obviously open to any feedback/guidance here however my main questions are:

Any suggestions would be greatly appreciated.

Traceback: I tried to post the full traceback here as well however including that caused the post to get flagged as 'this looks like spam'

Internal Server Error: /build/
Traceback (most recent call last):
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 49, in _reraise_errors
    yield
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 220, in dumps
    payload = encoder(data)
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 65, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 55, in default
    return super().default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Template is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)

Upvotes: 5

Views: 4775

Answers (2)

SomeRandomDude111
SomeRandomDude111

Reputation: 31

I had similar problem and found out two solutions:

  1. OP's first approach
# view
...
orders_json = serializers.serialize('json', orders)
cancel_order_task.delay(orders_json)
...

# task
def resend_order_task(orders_json: dict):
    orders = list(serializers.deserialize('json', orders_json))
    for order in orders:
        order = order.object
...

Key difference from OP's approach is order = order.object. It allows you to access attributes of an object naturally: order.id, order.params, etc.

Pros:

  • You send an entire object (just how OP wanted to initially)
  • Less operations with DB

Cons:

  • Cost of (de)serialization scale linearly with object size and complexity and might slow down performance for larger and more complex objects
  • Havier messages to send to celery
  1. Raihan K. suggestion
# view
...
order_ids = [order.id for order in orders]
resend_order_task.delay(order_ids)
...

# task
@shared_task
def resend_order_task(order_ids: list):
    orders = Order.objects.filter(id__in=order_ids)
    for order in orders:
...

Pros:

  • No compute power for (de)serializing
  • Less traffic needed to send task
  • You get an actual object from DB and can manipulate it right away

Cons:

  • More DB operations required

Upvotes: 0

Raihan K.
Raihan K.

Reputation: 591

Add this lines to settings.py

# Project/settings.py
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Then instead of passing object, send JSON with id/pk if you're using a model instance call the task like this..

test.delay({'pk': 1})

Django model instance is not available in celery environment, as it runs in a different process

How you can get the model instance inside task then? Well, you can do something like below -

def import_django_instance():
    """
    Makes django environment available 
    to tasks!!
    """
    import django
    import os
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Project.settings')
    django.setup()


# task
@shared_task(name="simple_task")
def simple_task(data):
    import_django_instance()
    from app.models import AppModel

    pk = data.get('pk')
    instance = AppModel.objects.get(pk=pk)
    # your operation

Upvotes: 9

Related Questions