CODEWITHSUNDEEP
pythondjangopython-multiprocessing
Sunil Vikram
Sunil Vikram

Reputation: 39

Django: running a few tasks in parallel

I have a web application build out of Django version 2.0.1

A user uploads a file, and based on the content, there are tasks which are executed in a serial fashion. After execution the results are shown to the user. Some of the tasks are independent of each other.

I want to execute the independent tasks in parallel. I tried using multiprocessing within views.py but there are some errors thrown when the processes are spawned. These tasks analyse some information and write to a file. The files are then combined to show the results to the user.

These tasks cannot be done asynchronous as the results produced needs to be shown to the user waiting. So I have dropped the idea of using Celery as recommended in other discussions.

Anyone's suggestions would be helpful.

Thanks

Error got

This was the error we gotTraceback (most recent call last): C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\multiprocessing\spawn.py", line 106, in spawn_main exitcode = _main(fd) File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\multiprocessing\spawn.py", line 116, in _main self = pickle.load(from_parent) File "G:\work\gitrepo\suprath-github\smartdata\ssd\FinalPlots\uploads\core\views.py", line 6, in from uploads.core.models import Document File "G:\work\gitrepo\suprath-github\smartdata\ssd\FinalPlots\uploads\core\models.py", line 7, in class Document(models.Model): File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\db\models\base.py", line 100, in new app_config = apps.get_containing_app_config(module) File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\apps\registry.py", line 244, in get_containing_app_config self.check_apps_ready() File "C:\Users\idea\AppData\Local\Enthought\Canopy\edm\envs\python\lib\site-packages\django\apps\registry.py", line 127, in check_apps_ready raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. Traceback (most recent call last):

Upvotes: 1

Views: 974

Answers (2)

sabikla
sabikla

Reputation: 91

I agree with the solution provided by @bruno desthuillieres, however, you can implement some socket solution to reach back to the user.

Since polling from user may have huge performance impacts, the socket solution will ideal for this case.

Upvotes: 0

bruno desthuilliers
bruno desthuilliers

Reputation: 77902

These tasks cannot be done asynchronous as the results produced needs to be shown to the user waiting

That doesn't mean you can't use an async queue (celery or other). We have a very similar use case and do use celery to run the tasks. The tasks (part parallel, part serial) store their progress in redis, and the frontend polls to get the current state and display progress to the user, then when the whole process is done (either successfuly or not) we display the result (or errors).

Upvotes: 2

Related Questions