CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
RodrikTheReader
RodrikTheReader

Reputation: 827

Spawning multiple processes with Python

Earlier I tried to use the threading module in python to create multiple threads. Then I learned about the GIL and how it does not allow taking advantage of multiple CPU cores on a single machine. So now I'm trying to do multiprocessing (I don't strictly need seperate threads).

Here is a sample code I wrote to see if distinct processes are being created. But as can be seen in the output below, I'm getting the same process ID everytime. So multiple processes are not being created. What am I missing? 

import multiprocessing as mp
import os

def pri():
    print(os.getpid())

if __name__=='__main__':

    # Checking number of CPU cores
    print(mp.cpu_count())

    processes=[mp.Process(target=pri()) for x in range(1,4)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

Output:

4
12554
12554
12554

Upvotes: 4

Views: 9484

Answers (2)

Chen A.
Chen A.

Reputation: 11280

Since the subprocesses runs on a different process, you won't see their print statements. They also don't share the same memory space. You pass pri() to target, where it needs to be pri. You need to pass a callable object, not execute it.

The prints you see are part of your main thread executions. Because you pass pri(), the code is actually executed. You need to change your code so the pri function returns value, rather than prints it.

Then you need to implement a queue, where all your threads write to it and when they're done, your main thread reads the queue.

A nice feature of the multiprocessing module is the Pool object. It allows you to create a thread pool, and then just use it. It's more convenient.

I have tried your code, the thing is the command executes too quick, so the OS reuses the PIDs. If you add a time.sleep(1) in your pri function, it would work as you expect.

That is True only for Windows. The example below is made on Windows platform. On Unix like machines, you won't need the sleep.

The more convenience solution is like this:

from multiprocessing import Pool
from time import sleep
import os

def pri(x):
    sleep(1)
    return os.getpid()

def use_procs():
    p_pool = Pool(4)
    p_results = p_pool.map(pri, [_ for _ in range(1,4)])
    p_pool.close()
    p_pool.join()
    return p_results

if __name__ == '__main__':
    res = use_procs()
    for r in res:
        print r

Without the sleep:

==================== RESTART: C:/Python27/tests/test2.py ====================
6576
6576
6576
>>>

with the sleep:

==================== RESTART: C:/Python27/tests/test2.py ====================
10396
10944
9000

Upvotes: 0

noxdafox
noxdafox

Reputation: 15040

The Process class requires a callable as its target.

Instead of running the function in the separate process, you are calling it and passing its result (None in this case) to the Process class.

Just change the following:

mp.Process(target=pri())

with:

mp.Process(target=pri)

Upvotes: 6

Related Questions