CODEWITHSUNDEEP
pythonmultiprocessingdillpathos
wye_naught
wye_naught

Reputation: 33

WindowsError in pathos.multiprocessing

import subprocess
import os
import dill
import pathos.multiprocessing as mp

def main(processes):
    list_of_files = os.listdir(os.getcwd())
    pool = mp.ProcessingPool(processes)
    pool.map(subprocess.call, [['program.exe',fname] 
        for fname in list_of_files])

if __name__ == "__main__":
    processes = 12
    main(processes)

Running the above gives the following error:

In [1]: %run batch.py
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
c:\3dns\v3.8\batch.py in <module>()
     26     processes = 12
---> 27     main(processes) 

c:\3dns\v3.8\batch.py in main(cores)
     19
---> 20     pool = mp.ProcessingPool(processes)

C:\Anaconda\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multiprocessing
.pyc in __init__(self, *args, **kwds)
     97
     98         # Create a new server if one isn't already initialized
---> 99         self._serve()
    100         return
    101     __init__.__doc__ = AbstractWorkerPool.__init__.__doc__ + __init__.__
doc__

C:\Anaconda\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multiprocessing
.pyc in _serve(self, nodes)
    108         _pool = __STATE['pool']
    109         if not _pool or nodes != _pool.__nodes:
--> 110             _pool = Pool(nodes)
    111             _pool.__nodes = nodes
    112             __STATE['pool'] = _pool

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\pool.pyc in __init__(self, processes, initializer, initargs)
     70     '''
     71     def __init__(self, processes=None, initializer=None, initargs=()):
---> 72         self._inqueue = SimpleQueue()
     73         self._outqueue = SimpleQueue()
     74         self._taskqueue = Queue.Queue()

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\queue.pyc in __init__(self)
    302
    303     def __init__(self):
--> 304         reader, writer = Pipe(duplex=False)
    305         if sys.platform == 'win32':
    306             state = reader, writer, Lock(), None

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\__init__.pyc in Pipe(duplex)
     95     '''
     96     from processing.connection import Pipe
---> 97     return Pipe(duplex)
     98
     99 def cpuCount():

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\connection.pyc in Pipe(duplex)
    230             win32.PIPE_TYPE_MESSAGE | win32.PIPE_READMODE_MESSAGE |
    231             win32.PIPE_WAIT,
--> 232             1, obsize, ibsize, win32.NMPWAIT_WAIT_FOREVER, win32.NULL
    233             )
    234         h2 = win32.CreateFile(

WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect

This was on a Windows 7 64-bit machine in the interactive IPython console (Anaconda Distribution of Python 2.7). I'm using pathos version 0.2a1.dev0 and Dill version 0.2.2. This error seems to be reproducible when ever I create a 'Pool'.

Upvotes: 0

Views: 610

Answers (2)

Mike McKerns
Mike McKerns

Reputation: 35217

I'm the pathos author. Indeed, it was @WillAngley suggested. The bug is now fixed. i have verified on windows 8.1, python 2.7, and with pathos and associated dependencies also in master or most recent release for 3-rd party. The critical issue was updating the fork of multiprocessing. This is done, and in a new package called multiprocess.

Upvotes: 1

Will Angley
Will Angley

Reputation: 1394

This looks a lot like a known bug in the underlying processing library when running on 64-bit Windows: pathos issue #49 on GitHub.

Have you tried using the Python multiprocessing library directly? It worked reliably for me on Windows when I used it last, although I don't have a Windows machine to hand to test specific behaviors at the moment.

Upvotes: 2

Related Questions