dkjackyu
dkjackyu

Reputation: 1

multiprocessing module not working in general on my computer

I would like to use multiprocessing to run different functions at the same time, by creating two different processes and starting them.

However, multiprocessing does not appear to work as expected on the particular machine on which I need to do this. It works on others, such Colab.

Here's what I'm using:

Another observation is that on the problematic machine, I can use joblib's Parallel to create a pool of processes, but not using multiprocessing.

It appears that multiprocessing is not working in general here in my current setup/machine? How can I diagnose this problem further? Thanks

Here are a few examples where it does not work as expected.

Example 1

This is an example from the official multiprocessing doc:

# Nothing printed from function `f` 

def f(name):
    print('hello', name)
    
p = Process(target=f, args=('bob',))
p.start()
p.join()

Example 2

Sleep statement in function does not appear to have run. The process appears to be NOT alive right after starting.

%%time

# Does not sleep.
# Process soon becomes NOT alive after starting

def f():
    time.sleep(10)
    
p = Process(target=f)
print(p.is_alive())
p.start()
print(p.is_alive())
p.join()
print(p.is_alive())

returns

False
True
False
CPU times: total: 31.2 ms
Wall time: 364 ms

Example 3

Using a pool, the executed cell just hangs, and the notebook needs to be restarted.

# Just hangs

def f(x):
    return x * x

with multiprocessing.Pool(5) as p:
    print(p.map(f, [1, 2, 3]))

Example 4

joblib works.

# joblib works

def f(x):
    return x * x

_f = delayed(f)
Parallel(n_jobs=5)(_f(x) for x in [1, 2, 3])

However, every example above works on other computers in notebook enviornments. They work in Colab, for example.

Upvotes: 0

Views: 47

Answers (0)

Related Questions