CODEWITHSUNDEEP
pythonpython-3.xmultiprocessing
BlueTrin
BlueTrin

Reputation: 10063

Multiprocessing pool in Python 3

I had a scripts working in Python 2 and I am trying to make it work in Python 3.

One thing I stumbled upon and have no idea about how to solve is the fact that the get() method in the class Applyresult() seems to throw now:

In Pycharm the Traceback is:

Traceback (most recent call last):
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 3.4.1\helpers\pydev\pydevd.py", line 1733, in <module>
    debugger.run(setup['file'], None, None)
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 3.4.1\helpers\pydev\pydevd.py", line 1226, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 3.4.1\helpers\pydev\_pydev_execfile.py", line 38, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc) #execute the script
  File "D:/SRC/CDR/trunk/RegressionTests/ExeTests/pyQCDReg_Launcher.py", line 125, in <module>
    result_list = ar.get()
  File "C:\Python33\lib\multiprocessing\pool.py", line 564, in get
    raise self._value
TypeError: can't use a string pattern on a bytes-like object

The line offending is in pool.py:

def get(self, timeout=None):
    self.wait(timeout)
    if not self.ready():
        raise TimeoutError
    if self._success:
        return self._value
    else:
        raise self._value    // This is the line raising the exception

It is called from the following line in my script:

pool = Pool(processes=8)
ar = pool.map_async(run_regtest, command_list)
pool.close()
start_time = time.time()
while True:
    elapsed = time.time() - start_time
    if (ar.ready()): break
    remaining = ar._number_left
    print("Elapsed,", elapsed, ", Waiting for", remaining, "tasks to complete...")
    time.sleep(30)

pool.join()
ar.wait()
print("finished")
result_list = ar.get() // This is the offending line causing the exception

This script was working in Python 2 and I cannot understand why it would not be working in Python 3. Does anyone have an idea why ?

Upvotes: 0

Views: 1243

Answers (1)

Andr&#233; Laszlo
Andr&#233; Laszlo

Reputation: 15537

From the multiprocessing documentation:

get([timeout])

Return the result when it arrives. If timeout is not None and
the result does not arrive within timeout seconds then
multiprocessing.TimeoutError is raised. If the remote call
raised an exception then that exception will be reraised by
get().

It seems likely to me that your exception comes from run_regtest.

The exception you are getting is pretty common when you switch from Python 2 to Python 3. Many functions in the standard library (and other libraries) that used to return strings now return bytes. A bytes object can be converted to a string using b.decode('utf-8'), for example, you only need to know the encoding.

Upvotes: 2

Related Questions