CODEWITHSUNDEEP
djangomultithreadingasynchronousconcurrencycelery
harold3sigma
harold3sigma

Reputation: 31

Django concurrency with celery

I am using django framework and ran into some performance problems.

There is a very heavy (which costs about 2 seconds) in my views.py. And let's call it heavy().

The client uses ajax to send a request, which is routed to heavy(), and waits for a json response.

The bad thing is that, I think heavy() is not concurrent. As shown in the image below, if there are two requests routed to heavy() at the same time, one must wait for another. In another word, heavy() is serial: it cannot take another request before returning from current request. The observation is tested and proven on my local machine.

synchronous django

I am trying to make the functions in views.py concurrent and asynchronous. Ideally, when there are two requests coming to heavy(), heavy() should throw the job to some remote worker with a callback, and return. Then, heavy() can process another request. When the task is done, the callback can send the results back to client. The logic is demonstrated as below:

asynchronous django

However, there is a problem: if heavy() wants to process another request, it must return; but if it returns something, the django framework will send a (fake)response to the client, and the client may not wait for another response. Moreover, the fake response doesn't contain the correct data. I have searched throught stackoverflow and find less useful tips. I wonder if anyone have tried this and knows a good way to solve this problem.

Thanks,

Upvotes: 1

Views: 500

Answers (2)

dm03514
dm03514

Reputation: 55952

To follow up on GwynBliedD's answer:

celery is commonly used to process tasks, it has very simple django integration. @GwynBlieD's first suggestion is very commonly implemented using celery and a celery result backend.

https://www.reddit.com/r/django/comments/1wx587/how_do_i_return_the_result_of_a_celery_task_to/

A common workflow Using celery is:

  1. client hits heavy()
  2. heavy() queues heavy() task asynchronously
  3. heavy() returns future task ID to client (view returns very quickly because little work was actually performed)
  4. client starts polling a status endpoint using the task ID
  5. when task completes status returns result to client

Upvotes: 0

GwynBleidD
GwynBleidD

Reputation: 20539

First make sure that 'inconcurrency' is actually caused by your heavy task. If you're using only one worker for django, you will be able to process only one request at a time, no matter what it will be. Consider having more workers for some concurrency, because it will affect also short requests.

For returning some information when task is done, you can do it in at least two ways:

  • sending AJAX requests periodicaly to fetch status of your task
  • using SSE or websocket to subscribe for actual result

Both of them will require to write some more JavaScript code for handling it. First one is really easy achievable, for second one you can use uWSGI capabilities, as described here. It can be handled asynchronously that way, independently of your django workers (django will just create connection and start task in celery, checking status and sending it to client will be handled by gevent.

Upvotes: 2

Related Questions