CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
top bantz
top bantz

Reputation: 615

Python Multiprocessing not running simultaneously

Let's say I have this simple function:

def print_name(name):
    print(name)
    time.sleep(10)
    print('finished sleeping @ ', str(dt.datetime.now()))

I'm trying to use multiprocessing to run this in a loop for several names all at once, like in the below:

from multiprocessing.pool import ThreadPool as Pool

names = ['A','B','C','D','E']
with Pool() as pool:
    for name in names:
        pool.map(print_name,[name])

However, it doesn't run simultaneously, it runs one after the other as you can see:

A
finished sleeping @  2022-07-26 11:03:12.394843
B
finished sleeping @  2022-07-26 11:03:22.400343
.......

NB: I'm having to use ThreadPool instead of Pool as Pool just throws up a random pickling error: _pickle.PicklingError: Can't pickle <function print_name at 0x144ce5000>: attribute lookup print_name on __main__ failed

I have also seen people talking about pathos.multiprocessing but that throws up this error NotImplementedError: pool objects cannot be passed between processes or pickled.

ThreadPool is the only way I can get any form of multiprocessing to at least not throw in an error message. So possibly that's my problem?

Ultimately I'm hoping to be able to use multiprocessing as I have a big function that takes about 15 mins to run, and I need to run it over a list of about 100 items so multiprocessing would be really handy. But I can't even get this simple example to work at the minute and I'm a bit stuck, so any help would be really appreciated.

Upvotes: 1

Views: 889

Answers (1)

Adon Bilivit
Adon Bilivit

Reputation: 27424

You may want to consider using ProcessPoolExecutor from concurrent.futures but in the meantime, this may help:

import datetime as dt
import time
from multiprocessing import Pool

def print_name(name):
    print(name)
    time.sleep(5)
    print('finished sleeping @ ', dt.datetime.now())

names = ['A','B','C','D','E']

def main():
    with Pool() as pool:
        pool.map(print_name, names)

if __name__ == '__main__':
    main()

Output:

A
B
C
D
E
finished sleeping @  2022-07-26 11:29:31.277722
finished sleeping @  2022-07-26 11:29:31.277749
finished sleeping @  2022-07-26 11:29:31.285708
finished sleeping @  2022-07-26 11:29:31.292636
finished sleeping @  2022-07-26 11:29:31.295505

Upvotes: 2

Related Questions