CODEWITHSUNDEEP
pythoniteratormultiprocessingpool
sten
sten

Reputation: 7476

Multiprocessing : Pool : Direct iterator OR use of variable to store the iterator

Is there are difference between :

  with Pool(...) as pool :
      res = pool.imap_unordered(fun, iterator)
      for val in res :
           ........

and ..

  with Pool(...) as pool :
      for val in pool.imap_unordered(fun, iterator) :
           ........

is the second variant faster !

Upvotes: 0

Views: 88

Answers (1)

Booboo
Booboo

Reputation: 44108

Here is the disassembled code of the two cases:

from multiprocessing import Pool

def fun(x):
    pass


def method1(it):
    with Pool() as pool:
        res = pool.imap_unordered(fun, it)
        for val in res:
            pass

def method2(it):
    with Pool() as pool:
        for val in pool.imap_unordered(fun, it):
            pass

import dis
dis.dis(method1)
print()
print('-' * 80)
print()
dis.dis(method2)

Prints:

  8           0 LOAD_GLOBAL              0 (Pool)
              2 CALL_FUNCTION            0
              4 SETUP_WITH              28 (to 34)
              6 STORE_FAST               1 (pool)

  9           8 LOAD_FAST                1 (pool)
             10 LOAD_METHOD              1 (imap_unordered)
             12 LOAD_GLOBAL              2 (fun)
             14 LOAD_FAST                0 (it)
             16 CALL_METHOD              2
             18 STORE_FAST               2 (res)

 10          20 LOAD_FAST                2 (res)
             22 GET_ITER
        >>   24 FOR_ITER                 4 (to 30)
             26 STORE_FAST               3 (val)

 11          28 JUMP_ABSOLUTE           24
        >>   30 POP_BLOCK
             32 BEGIN_FINALLY
        >>   34 WITH_CLEANUP_START
             36 WITH_CLEANUP_FINISH
             38 END_FINALLY
             40 LOAD_CONST               0 (None)
             42 RETURN_VALUE

--------------------------------------------------------------------------------

 14           0 LOAD_GLOBAL              0 (Pool)
              2 CALL_FUNCTION            0
              4 SETUP_WITH              24 (to 30)
              6 STORE_FAST               1 (pool)

 15           8 LOAD_FAST                1 (pool)
             10 LOAD_METHOD              1 (imap_unordered)
             12 LOAD_GLOBAL              2 (fun)
             14 LOAD_FAST                0 (it)
             16 CALL_METHOD              2
             18 GET_ITER
        >>   20 FOR_ITER                 4 (to 26)
             22 STORE_FAST               2 (val)

 16          24 JUMP_ABSOLUTE           20
        >>   26 POP_BLOCK
             28 BEGIN_FINALLY
        >>   30 WITH_CLEANUP_START
             32 WITH_CLEANUP_FINISH
             34 END_FINALLY
             36 LOAD_CONST               0 (None)
             38 RETURN_VALUE

Results

It would appear that the method1 results in executing two more bytecode instructions (at locations 18 and 20) regardless of the length of the iterator or the amount of processing done by fun or in processing the results returned by fun. In other words, such a negligible difference for there to be any cause for concern.

Upvotes: 1

Related Questions