Reputation: 1
I would like to use multiprocessing
to run different functions at the same time, by creating two different processes and starting them.
However, multiprocessing
does not appear to work as expected on the particular machine on which I need to do this. It works on others, such Colab.
Here's what I'm using:
Another observation is that on the problematic machine, I can use joblib
's Parallel
to create a pool of processes, but not using multiprocessing
.
It appears that multiprocessing
is not working in general here in my current setup/machine? How can I diagnose this problem further? Thanks
Here are a few examples where it does not work as expected.
This is an example from the official multiprocessing doc:
# Nothing printed from function `f`
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
Sleep statement in function does not appear to have run. The process appears to be NOT alive right after starting.
%%time
# Does not sleep.
# Process soon becomes NOT alive after starting
def f():
time.sleep(10)
p = Process(target=f)
print(p.is_alive())
p.start()
print(p.is_alive())
p.join()
print(p.is_alive())
returns
False
True
False
CPU times: total: 31.2 ms
Wall time: 364 ms
Using a pool, the executed cell just hangs, and the notebook needs to be restarted.
# Just hangs
def f(x):
return x * x
with multiprocessing.Pool(5) as p:
print(p.map(f, [1, 2, 3]))
joblib
works.
# joblib works
def f(x):
return x * x
_f = delayed(f)
Parallel(n_jobs=5)(_f(x) for x in [1, 2, 3])
However, every example above works on other computers in notebook enviornments. They work in Colab, for example.
Upvotes: 0
Views: 47