CODEWITHSUNDEEP
pythonprocessparallel-processingmultiprocessingpool
ss301
ss301

Reputation: 672

multiprocessing in python does not stop running

I tried a simple example of multiprocessing in python from their website itself, but it does not give any input. It's showing as running itself and I am not able to stop it in jupyter notebook.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1, 2, 3]))

It's the same for other multiprocessing examples too. It does not give any error or timeout or anything. It's like it is in an infinite loop or deadlock.

Upvotes: 3

Views: 4972

Answers (2)

Pieter Van Bogaert
Pieter Van Bogaert

Reputation: 76

I'm also on Windows.

  1. As pointed out 'from multiprocessing.pool import ThreadPool as Pool' works.
  2. also you need to add if name == 'main':
  3. also below that you need to add : 'with Pool(4) as pool:'

example

from multiprocessing.pool import ThreadPool as Pool
from os import getpid
import time
import pandas as pd


pyfiles = [10,2,3,5]    

def scraper(x):
    results_df = pd.DataFrame({})
    print('Program started:',x,"I'm process", getpid())
    time.sleep(x)
    print('Program completed:',x)
    results_df.to_csv('multi{}.csv'.format(x))


if __name__ == '__main__':
    with Pool(4) as pool:
        start=time.time()
        result = pool.map(scraper, pyfiles)
        pool.terminate()
        pool.join()
        print("Time Taken: ",str(time.time()-start))

Upvotes: 6

ss301
ss301

Reputation: 672

I dont know how but it worked when i imported like given below.

from multiprocessing.pool import ThreadPool as Pool

The problem was with pool in importing multiprocessing.

Upvotes: 4

Related Questions