CODEWITHSUNDEEP
pythondjangomultithreadingpython-multiprocessingpython-multithreading
Saurabh_Jhingan
Saurabh_Jhingan

Reputation: 320

Why do I need to use asynchronous tools for sending Django email

While using Django, I had noticed when I send an ​email there is a delay and to overcome this I had to use Celery tool. So I wanted to know what is actually happening under the hood. Why does Django/Python need an async tool to accomplish this task. I have learned about MultiThreading, MultiProcessing in Python but if somebody could give me a brief idea about what exactly is happening when Django is trying to send an email without using Celery.

Upvotes: 1

Views: 486

Answers (1)

A. J. Parr
A. J. Parr

Reputation: 8026

Think of sending an email like sending a request, in a synchronous context the process would go as follows:

  1. Send request
  2. Wait for response..........
  3. Receive response

The whole time you're waiting for the response that thread cannot do anything else, it's wasting CPU cycles that could be used by something else (such as serving other users requests).

I'd like to make a distinction here between your usage of asynchronous and celery. Pythons actual asynchronous implementation uses an "event loop" to dispatch and receive messages. The "waiting" is done in a separate thread/process which is used exclusively to receive messages, and dispatch those messages to receivers, in this way your thread that sent the request is no longer wasting CPU cycles waiting, it will be invoked by the event loop when it's ready. This is a very vague description of how pythons async works. It won't necessarily make the whole process faster for the user unless there are a lot of emails being sent.

Celery on the other hand is an asynchronous task queue, in which you have producers (your web application) sending messages, a broker (data store) which stores and distributes messages, and consumers (workers) which pull messages from the broker and processes them. The consumers are a totally separate process (often a totally separate server) from your web application, it frees up your web application to focus on returning the response to the client as soon as possible. The process when sending emails through celery would look like:

  1. Web application sends a message to the broker and returns the response to the user. Here's a json pseudo-message. (The broker actually stores the messages as either pickled objects or JSON)
{
    "task": "my_app.send_email",
    "args": ["Subject Line", "Hello, World! This is your email contents", "[email protected]", "[email protected]"],  # 
    "kwargs": {}  # No keyword arguments
}
  1. The celery worker is constantly checking with the broker for new messages to process if it is not currently processing. Sometimes the celery worker will pull in batches of messages so there is less overhead, this is configurable.
  2. The celery worker executes a function (defined by the "task" in the message), using the arguments and keyword arguments.

That is a very simple example of why you may want to use celery to send emails, so you can return the response to the user as fast as possible! It's also well suited to longer running tasks, such as processing image thumbnails:

  1. User uploads an image, which you store somewhere (Amazon S3 for example)
  2. You send a message to the broker saying "execute my process_image_thumbails task with the files S3 URL as the argument"
  3. You return the response to your user. It's nice and quick from the users perspective.
  4. A worker picks up the message, downloads the file from S3, and processes it into thumbnails of varying sizes.

As you use celery for more new use cases you encounter new problems. For example, what do we do if someone requests the thumbnail while it's processing? I'll leave that to your imagination.

Upvotes: 2

Related Questions