CODEWITHSUNDEEP
pythonasynchronousmultiprocessingthreadpool
Mahdi
Mahdi

Reputation: 1852

Can I pass a method to apply_async or map in python multiprocessing?

How can I get the following to work? The main point is that I want to run a method (and not a function) asynchronously.

from multiprocessing import Pool

class Async:
    def __init__(self, pool):
        self.pool = pool
        self.run()

    def run(self):
        p.apply_async(self.f, (10, ))

    def f(self, x):
        print x*x

if __name__ == '__main__':
    p = Pool(5)
    a = Async(p)
    p.close()
    p.join()

This prints nothing.

Upvotes: 1

Views: 5227

Answers (2)

Mahdi
Mahdi

Reputation: 1852

The problem appears to be due to the fact that multiprocessing needs to pickle self.f while bound methods are not picklable. There is a discussion on how to solve the problem here.

The apply_async apparently creates an exception which is put inside the future returned. That's why nothing is printed. If a get is executed on the future, then the exception is raised.

Upvotes: 3

frankster
frankster

Reputation: 1528

Its definitely possible to thread class methods using a threadpool in python 2 - the following programme did what I would expect.

#!/usr/bin/env python

from multiprocessing.pool import ThreadPool

class TestAsync():
  def __init__(self):
    pool = ThreadPool(processes = 2)

    async_completions = []
    for a in range(2):
      async_completions.append(pool.apply_async(self.print_int, (  a,)))

    for completion in async_completions:
      res = completion.get()
      print("res = %d" % res)

  def print_int(self, value):
    print(value)
    return (value*10)


a = TestAsync()

Upvotes: 0

Related Questions