How to Send Asynchronous Emails In Django Using Celery?

Question

I'm trying to send asynchronous emails in a Django 1.6.2 application using django-celery-email and Celery 3.1.17 with RabbitMQ 3.5.0 as my message broker and result backend. Emails are being sent and received but I also get the error "django.core.mail... is not JSON serializable" in my Celery log when emails are sent. I'm using JSON for serialization in Celery as pickle is being deprecated. Is there some way I can change my configuration to prevent this error from occurring? Incidentally, I can send emails just fine when I don't use Celery's delay method.

Thanks.

# Stacktrace
Task app.tasks.send_email with id 7ac1eb8e-c090-4893-8147-1f204e463d12 raised exception:
'EncodeError(TypeError("<module \'django.core.mail\' from \'/Users/me/venv/django/lib/python2.7/site-packages/django/core/mail/__init__.pyc\'> is not JSON serializable",),)'


Task was called with args: [] kwargs: {}.

The contents of the full traceback was:

Traceback (most recent call last):
 File "/Users/me/venv/django/lib/python2.7/site-packages/celery/app/trace.py", line 283, in trace_task
   uuid, retval, SUCCESS, request=task_request,
 File "/Users/me/venv/django/lib/python2.7/site-packages/celery/backends/amqp.py", line 136, in store_result
   delivery_mode=self.delivery_mode,
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/messaging.py", line 161, in publish
   compression, headers)
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/messaging.py", line 237, in _prepare
   body) = dumps(body, serializer=serializer)
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
   payload = encoder(data)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
   self.gen.throw(type, value, traceback)
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/serialization.py", line 59, in _reraise_errors
   reraise(wrapper, wrapper(exc), sys.exc_info()[2])
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/serialization.py", line 55, in _reraise_errors
   yield
 File "/Users/me/venv/django/lib/python2.7/site-packages/kombu/serialization.py", line 164, in dumps
   payload = encoder(data)
 File "/Users/me/venv/django/lib/python2.7/site-packages/anyjson/__init__.py", line 141, in dumps
   return implementation.dumps(value)
 File "/Users/me/venv/django/lib/python2.7/site-packages/anyjson/__init__.py", line 87, in dumps
   return self._encode(data)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
   return _default_encoder.encode(obj)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
   chunks = self.iterencode(o, _one_shot=True)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
   return _iterencode(o, 0)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
   raise TypeError(repr(o) + " is not JSON serializable")
EncodeError: <module 'django.core.mail' from '/Users/me/venv/django/lib/python2.7/site-packages/django/core/mail/__init__.pyc'> is not JSON serializable

# settings.py
# MAIL SETTINGS
EMAIL_HOST = 'smtp.sendgrid.net'
EMAIL_HOST_USER = 'me'
EMAIL_HOST_PASSWORD ='password'
EMAIL_PORT = 587
EMAIL_USE_TLS = True
FAIL_SILENTLY = False

# CELERY SETTINGS
BROKER_URL = 'amqp://guest@localhost//'
CELERY_RESULT_BACKEND = 'amqp'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json', ]

# DJANGO-CELERY-EMAIL
INSTALLED_APPS += ('djcelery_email', )
EMAIL_BACKEND = 'djcelery_email.backends.CeleryEmailBackend'

# tasks.py
from __future__ import absolute_import

import smtplib
from conf.celery import app
from django.core import mail
from django.template.loader import render_to_string

@app.task
def send_email():
    to_email = 'me@example.com'
    subject = 'Testing Celery/RabbitMQ'
    from_email = 'me2@example.com'
    message = 'This is a test of my Celery/RabbitMQ function.'
    recipient_list = []
    recipient_list.append(to_email)
    html_message = render_to_string('send_mail.html', {'message': message})
    try:
        mail.send_mail(subject, message, from_email, recipient_list, html_message)
    except smtplib.SMTPException, e:
        return 0
    return mail

# views.py
from app.tasks import send_email
def home_page(request, template):
    # Send mail synchronously
    #send_email()
    # Send email asynchronously.
    send_email.delay()
    return render(request, template)

Answer 1

You're getting the error as expected. Celery is setup to use JSON serialization and is using the built in json library that attempts to serialize django.core.mail which obviously doesn't support any form of serialization.

The reason it works when calling without delay is because it works like calling a typical function (in the same process).

If you have to return something from a task, you can return a dictionary of values or objects that support serialization. For custom serializations, you can use the serializer property and pass your custom decoder/encoder when calling the task.

In your case you can return something like {'success': True} and False value for any failures.

More options on using custom serializers