CODEWITHSUNDEEP
pythonpython-multiprocessing
usual me
usual me

Reputation: 8788

Can I use map / imap / imap_unordered with functions with no arguments?

Sometimes I need to use multiprocessing with functions with no arguments. I wish I could do something like:

from multiprocessing import Pool

def f():  # no argument
    return 1

# TypeError: f() takes no arguments (1 given)
print Pool(2).map(f, range(10))

I could do Process(target=f, args=()), but I prefer the syntax of map / imap / imap_unordered. Is there a way to do that?

Upvotes: 25

Views: 17186

Answers (4)

Sara
Sara

Reputation: 537

For using the method pool, from library multiprocessing, you should provide an argument. Thus, you should use another method like Process from multiprocessing:

if __name__ == '__main__':
    p = Process(target=f)
    p.start()
    p.join()

Upvotes: 0

LearnOPhile
LearnOPhile

Reputation: 1561

You can use pool.starmap() instead of .map() like so:

from multiprocessing import Pool

def f():  # no argument
    return 1

print Pool(2).starmap(f, [() for _ in range(10)])

starmap will pass all elements of the given iterables as arguments to f. The iterables should be empty in your case.

Upvotes: 19

Dunes
Dunes

Reputation: 40833

Is there anything wrong with using Pool.apply_async?

with multiprocessing.Pool() as pool:
    future_results = [pool.apply_async(f) for i in range(n)]
    results = [f.get() for f in future_results]

Upvotes: 3

thefourtheye
thefourtheye

Reputation: 239633

map function's first argument should be a function and it should accept one argument. It is mandatory because, the iterable passed as the second argument will be iterated and the values will be passed to the function one by one in each iteration.

So, your best bet is to redefine f to accept one argument and ignore it, or write a wrapper function with one argument, ignore the argument and return the return value of f, like this

from multiprocessing import Pool

def f():  # no argument
    return 1

def throw_away_function(_):
    return f()

print(Pool(2).map(throw_away_function, range(10)))
# [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

You cannot use lamdba functions with pools because they are not picklable.

Upvotes: 14

Related Questions