CODEWITHSUNDEEP
pythonmultiprocessing
gatoatigrado
gatoatigrado

Reputation: 16850

yet another confusion with multiprocessing error, 'module' object has no attribute 'f'

I know this has been answered before, but it seems that executing the script directly "python filename.py" does not work. I have Python 2.6.2 on SuSE Linux.

Code:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool
p = Pool(1)
def f(x):
    return x*x
p.map(f, [1, 2, 3])

Command line:

> python example.py
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap
    self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 339, in get
    return recv()
AttributeError: 'module' object has no attribute 'f'

Upvotes: 73

Views: 44912

Answers (5)

Yonas Kassa
Yonas Kassa

Reputation: 3690

This comes from the fact that with p = Pool(1) the main process forks processes (threads vs processes) before it creates the function f. As stated in Bartosz answer the spawned processes do not have access to the new function.

def f1(x):
    ...

p = Pool(1) # p is spawned and is now an independent process, knows f1

def f(x): # p doesn't not share this object
    ...

Upvotes: 1

Pluto
Pluto

Reputation: 854

The problem I had was solved by using if __name__ == "__main__" as pointed out by Tamás; in Eclipse for Windows the examples do not work under the interpreter. This is explained in http://docs.python.org/2/library/multiprocessing

Upvotes: 1

user1143523
user1143523

Reputation: 79

One possibility is that your python file has the same name as a module:

  • test.py
  • test/
    • __init__.py

in pickle.py, you have the error coming from:

    def find_class(self, module, name):
      # Subclasses may override this
      __import__(module)
      mod = sys.modules[module] # <- here mod will reference your test/__init__.py
      klass = getattr(mod, name)
      return klass

Upvotes: 1

Bartosz
Bartosz

Reputation: 6193

Restructure your code so that the f() function is defined before you create instance of Pool. Otherwise the worker cannot see your function.

#!/usr/bin/python
# -*- coding: utf-8 -*-

from multiprocessing import Pool

def f(x):
    return x*x

p = Pool(1)
p.map(f, [1, 2, 3])

Upvotes: 138

Tam&#225;s
Tam&#225;s

Reputation: 48061

This one works:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == "__main__":
    p = Pool(1)
    p.map(f, [1, 2, 3])

I'm not 100% sure why your code does not work, but I guess the reason is that child processes launched by the multiprocessing module try to import the main module (to have access to the methods you defined), and the if __name__ == "__main__" stanza is required not to execute the initialization code where you set up your pool.

Upvotes: 7

Related Questions