CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingpython-multiprocessing
Bondrak
Bondrak

Reputation: 1590

multiprocessing in python: AttributeError

I'm trying to use multiprocessing, but I keep getting this error: 

AttributeError: Can't get attribute 'processLine' on <module '__main__'

(The processLine function returns word, so I guess the problem is here, but I don't know how to get around it)

import multiprocessing as mp

pool = mp.Pool(4)
jobs = []
Types =[]

def processLine(line):
     line = line.split()
     word = line[0].strip()
     return word

with open("1.txt", "r", encoding = "utf-8") as f:
     for line in f:
          word = (jobs.append(pool.apply_async(processLine,(line))))
          Types.append(word)
     filtered_words=[]
     with open("2.txt", "r", encoding = "utf-8") as f:
          for line in f:
               word = jobs.append(pool.apply_async(processLine,(line)))
               if word in Types:
                    filtered_words = "".join(line)
     print(filtered_words)

for job in jobs:
    job.get()

pool.close()

And this is what I get: Process ForkPoolWorker-1:

Process ForkPoolWorker-2:

Process ForkPoolWorker-3:

Process ForkPoolWorker-4:

Traceback (most recent call last):

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap

self.run()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker

task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res)

AttributeError: Can't get attribute 'processLine' on

Traceback (most recent call last):

Traceback (most recent call last):

Traceback (most recent call last):

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap

self.run()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker

task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get() File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res) File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res) File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) AttributeError: Can't get attribute 'processLine' on AttributeError: Can't get attribute 'processLine' on File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get

return _ForkingPickler.loads(res)

AttributeError: Can't get attribute 'processLine' on

Upvotes: 0

Views: 4080

Answers (3)

Jaswanth Alla
Jaswanth Alla

Reputation: 1

I faced the same issue. So, I put all functions in another .py file and imported ipnyb, and called them inside the main function.

Upvotes: 0

Vince P.
Vince P.

Reputation: 509

I worked around the AttributeError issue by using VS Code in administrator mode to run it instead of Anaconda Spyder.

Upvotes: 0

larsks
larsks

Reputation: 311615

The multiprocessing module needs to be able to import your module safely. Any code not inside a function or class should be protected by the standard Python import guard:

if __name__ == '__main__':
    ...code goes here...

But there are other problems with your code. For example, you've got:

word = jobs.append(pool.apply_async(processLine,(line)))

...but append doesn't return a value, so this will always assign None to word.

Rather than using a for loop to repeatedly call pool.apply_async, you may want to consider using pool.map_async instead, or just pool.map if you don't actually need the asynchronous behavior.

Upvotes: 4

Related Questions