CODEWITHSUNDEEP
pythonmultiprocessing
Cheng-Yu Dai
Cheng-Yu Dai

Reputation: 23

multiprocessing imap suprress child process print

While I'm running the following code in Jupyter notebook (python3.5):

def process(doc):
    print('Process Hey!')
with Pool() as pool:
    results = pool.imap(process, docs)

This would not give the print to the notebook, but if I try using pool.map instead, it will spit out the words. So I'm really curious what's going on up here. Thanks!

Upvotes: 2

Views: 476

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180482

From the docs imap A lazier version of map(), imap returns an iterator map returns a list:

In [24]:  with Pool() as pool:
            results = pool.map(process, docs)
            print(type(results))
   ....:     
Process Hey!
Process Hey!
<class 'list'>


In [25]:  with Pool() as pool:
                results = pool.imap(process, docs)
                print(type(results))
   ....:     
<class 'multiprocessing.pool.IMapIterator'>

In [27]:  with Pool() as pool:
                results = pool.imap(process, docs)
                for _ in results:
                     pass
   ....:     
Process Hey!
Process Hey!
In [28]:  with Pool() as pool:
            results = pool.imap(process, docs)
            list(results)
 ....:     
Process Hey!
Process Hey!

It is equivalent to the difference between itertools.imap and map in python2 i.e lazily evaluated vs greedily evaluated:

In [3]: from itertools import imap

In [4]: r = imap(lambda x: x+1, [1,2,3])
In [4]: r
Out[5]: <itertools.imap at 0x7f09170f1e10>
In [6]: list(r)
Out[6]: [2, 3, 4]

In [7]: r = map(lambda x: x+1, [1,2,3])    
In [8]: r
Out[8]: [2, 3, 4]

Upvotes: 1

Related Questions