CODEWITHSUNDEEP
pythondjangoasynchronouscelerydjango-celery
hR 312
hR 312

Reputation: 854

How can we create asynchronous API in django?

I want to create a third party chatbot API which is asynchronous and replies "ok" after 10 seconds pause.

import time

def wait():
    time.sleep(10)
    return "ok"

# views.py
def api(request):
    return wait()

I have tried celery for the same as follows where I am waiting for celery response in view itself:

import time
from celery import shared_task

@shared_task
def wait():
    time.sleep(10)
    return "ok"

# views.py
def api(request):
    a = wait.delay()
    work = AsyncResult(a.id)
    while True:
        if work.ready():
           return work.get(timeout=1)

But this solution works synchronously and makes no difference. How can we make it asynchronous without asking our user to keep on requesting until the result is received?

Upvotes: 1

Views: 15137

Answers (3)

gelonida
gelonida

Reputation: 5640

As mentioned in @Blusky's answer: The asynchronous API will exist in django 3.X. not before.

If this is not an option, then the answer is just no.

Please note as well, that even with django 3.X any django code, that accesses the database will not be asynchronous it had to be executed in a thread (thread pool)

Celery is intended for background tasks or deferred tasks, but celery will never return an HTTP response as it didn't receive the HTTP request to which it should respond to. Celery is also not asyncio friendly.

You might have to think of changing your architecture / implementation. Look at your overall problem and ask yourself whether you really need an asynchronous API with Django.

Is this API intended for browser applications or for machine to machine applications?

Could your client's use web sockets and wait for the answer?

Could you separate blocking and non blocking parts on your server side? Use django for everything non blocking, for everything periodic / deferred (django + celelry) and implement the asynchronous part with web server plugins or python ASGI code or web sockets.

Some ideas

Use Django + nginx nchan (if your web server is nginx)

Link to nchan: https://nchan.io/ your API call would create a task id, start a celery task, return immediately the task id or a polling url.

The polling URL would be handled for example via an nchan long polling channel. your client connects to the url corresponding to an nchan long polling channel and celery deblocks it whenever you're task is finished (the 10s are over)

Use Django + an ASGI server + one handcoded view and use strategy similiar to nginx nchan

Same logic as above, but you don't use nginx nchan, but implement it yourself

Use an ASGI server + a non blocking framework (or just some hand coded ASGI views) for all blocking urls and Django for the rest.

They might exchange data via the data base, local files or via local http requests.

Just stay blocking and throw enough worker processes / threads at your server

This is probably the worst suggestion, but if it is just for personal use, and you know how many requests you will have in parallel then just make sure you have enough Django workers, so that you can afford to be blocking. I this case you would block an entire Django worker for each slow request.

Use websockets. e.g. with the channels module for Django

Websockets can be implemented with earlier versions of django (>= 2.2) with the django channels module (pip install channels) ( https://github.com/django/channels )

You need an ASGI server to server the asynchronous part. You could use for example Daphne ot uvicorn (The channel doc explains this rather well)

Addendum 2020-06-01: simple async example calling synchronous django code

Following code uses the starlette module as it seems quite simple and small

miniasyncio.py

import asyncio
import concurrent.futures
import os
import django
from starlette.applications import Starlette
from starlette.responses import Response
from starlette.routing import Route

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'pjt.settings')
django.setup()

from django_app.xxx import synchronous_func1
from django_app.xxx import synchronous_func2

executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)

async def simple_slow(request):
    """ simple function, that sleeps in an async matter """
    await asyncio.sleep(5)
    return Response('hello world')

async def call_slow_dj_funcs(request):
    """ slow django code will be called in a thread pool """
    loop = asyncio.get_running_loop()
    future_func1 = executor.submit(synchronous_func1)
    func1_result = future_func1.result()
    future_func2 = executor.submit(synchronous_func2)
    func2_result = future_func2.result()
    response_txt = "OK"
    return Response(response_txt, media_type="text/plain")

routes = [
    Route("/simple", endpoint=simple_slow),
    Route("/slow_dj_funcs", endpoint=call_slow_dj_funcs),
]

app = Starlette(debug=True, routes=routes)

you could for example run this code with

pip install uvicorn
uvicorn --port 8002 miniasyncio:app

then on your web server route these specific urls to uvicorn and not to your django application server.

Upvotes: 9

A Khalili
A Khalili

Reputation: 222

Checkout Django 3 ASGI (Asynchronous Server Gateway Interface) support:
https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/

Upvotes: -2

Blusky
Blusky

Reputation: 3804

The best option is to use the futur async API, which will be proposed on Django in 3.1 release (which is already available in alpha)

https://docs.djangoproject.com/en/dev/releases/3.1/#asynchronous-views-and-middleware-support

(however, you will need to use an ASGI Web Worker to make it work properly)

Upvotes: 0

Related Questions