CODEWITHSUNDEEP
djangoparallel-processingconcurrent.futuresdjango-4.1
Poorya
Poorya

Reputation: 11

How use ProcessPoolExecutor in django?

In a Django Project I want to use ProcessPoolExecutor but I can do that. Is there any solution? I call a function in view that has executor.

I use Django==4.1.7.

what should add to my function to run in django. The start() function works properly outside django. but doesn't work in django.

from a views.py in django app call a function that contain executor:

def start():
     #'... some code here'

     with concurrent.futures.ProcessPoolExecutor(max_workers=num_procs) as executor:

Upvotes: 0

Views: 255

Answers (2)

Shaheed Haque
Shaheed Haque

Reputation: 723

Generally, both Python ProcessPoolExecutor, and multiprocessing on which it is based, need the start() to be usable when the module containing it is instantiated by itself (as opposed to being instantiated as part of you larger program). This needs:

  1. The function to be usable from module scope (as your start() already is; a @staticmethod in a class will also work just fine).
  2. The module to be importable by itself.

Somewhat opaquely, this latter bit is mentioned in the docs which say:

The __main__ module must be importable by worker subprocesses.

Now, on Linux with Python prior to 3.13, this seems to work without any extra effort because the default "start method" is fork, which actually works without this requirement. However, on MacOS and Windows and also Linux from Python 3.13, the default start method is/will be spawn which does have this as hard requirement.

Therefore, the safe answer is to actually follow the requirement stated in the docs. For many Django-based programs, this involves ensuring the Django initialisation is done in the module. This can be as simple as adding:

import django

django.setup()

More generally, one would need to try to import the module, and see what exception are thrown, and fix those.

Finally, it is worth noting that there are plenty of somewhat misleading articles on the web which talk about a mysterious need to "guard main" with a sequence such as:

def main():
    ...
    executor = ProcessPoolExecutor(...)


if __name__ == '__main__':
    main()

However, what they are really doing is making sure that when the module is imported in the worker subprocess, the call to ProcessPoolExecutor() is not performed. If your call to ProcessPoolExecutor() does not happen during module import, you don't need any of this magic.

Upvotes: 0

wishiknew
wishiknew

Reputation: 359

I recommend using Celery for calling out asynchronous tasks from Django. This way you will also have a way of tracking the process & possible errors. It takes a little to get to know the logic, but it's worth it.

https://docs.celeryq.dev/en/stable/index.html

https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html

Upvotes: 0

Related Questions