CODEWITHSUNDEEP
pythonconcurrencymultiprocessing
half of a glazier
half of a glazier

Reputation: 2076

Difference between multiprocessing and concurrent libraries?

Here's what I understand:

The multiprocessing library uses multiple cores, so it's processing in parallel and not just simulating parallel processing like some libraries. To do this, it overrides the Python GIL.

The concurrent library doesn't override the Python GIL and so it doesn't have the issues that multiprocessing has (ie locking, hanging). So it seems like it's not actually using multiple cores.

I understand the difference between concurrency and parallelism. My question is:

How does concurrent actually work behind the scenes?

And does subprocess work like multiprocessing or concurrent?

Upvotes: 2

Views: 585

Answers (2)

Tim Peters
Tim Peters

Reputation: 70582

multiprocessing and concurrent.futures both aim at running Python code in multiple processes concurrently. They're different APIs for much the same thing. multiprocessing's API was, as @András Molnár said, designed to be much like the threading module's. concurrent.futures's API was intended to be simpler.

Neither has anything to do with the GIL. The GIL is a per-process lock in CPython, and the Python processes these modules create each have their own GIL. You can't have a CPython process without a GIL, and there's no way to "override" it (although C code can release it when it's going to be running code that it knows for certain cannot execute Python code - for example, the CPython implementation routinely releases it internally when invoking a blocking I/O function in C, so that other threads can run Python code while the thread that released the GIL waits for the I/O call to complete).

Upvotes: 4

András Molnár
András Molnár

Reputation: 11

The subprocess module lets you run and control other programs. Anything you can start with the command line on the computer, can be run and controlled with this module. Use this to integrate external programs into your Python code.

The multiprocessing module lets you divide tasks written in python over multiple processes to help improve performance. It provides an API very similar to the threading module; it provides methods to share data across the processes it creates, and makes the task of managing multiple processes to run Python code (much) easier. In other words, multiprocessing lets you take advantage of multiple processes to get your tasks done faster by executing code in p

Upvotes: 1

Related Questions