CODEWITHSUNDEEP
pythonmultithreadingmultiprocessing
deflator
deflator

Reputation: 169

Python multiprocessing fails randomly and silently

I am a php developer trying to wing a python multiprocessing script. It is creating a queue that needs to be 40,000+ items long. The queue fills fine, but my processes die silently and randomly. Tried in 2.6 and 2.7.

I have put logging throughout to try to diagnose what is happening, to no avail. On random entries with no discernible data problems, all child processes will stop and the main thread will exit? Note: it is the SAME entries that it fails on, regardless of start point, unless it is under 20 entries being read.

Sometimes it will do 150 entries, sometimes 50 before it quits. Since I need it to do 40k, this is a deal-breaker. The script has no problem doing 20 or so, and logs the proper process starting and exiting messages. When it fails, there is not process exit logged.

My code is too sensitive to post, but here is the basic model I used, taken from a tutorial on multithreading and converted to the mutliprocessing module. Basically, just replaced "threading" with "multiprocessing" and used multiprocessing's Queue instead of the module Queue.

    import Queue
    import threading
    import time

    exitFlag = 0

    class myThread (threading.Thread):
        def __init__(self, threadID, name, q):
            threading.Thread.__init__(self)
            self.threadID = threadID
            self.name = name
            self.q = q
        def run(self):
            print "Starting " + self.name
            process_data(self.name, self.q)
            print "Exiting " + self.name

    def process_data(threadName, q):
        while not exitFlag:
            queueLock.acquire()
            if not workQueue.empty():
                data = q.get()
                queueLock.release()
                print "%s processing %s" % (threadName, data)
            else:
                queueLock.release()
            time.sleep(1)

    threadList = ["Thread-1", "Thread-2", "Thread-3"]
    nameList = ["One", "Two", "Three", "Four", "Five"]
    queueLock = threading.Lock()
    workQueue = Queue.Queue(10)
    threads = []
    threadID = 1

    # Create new threads
    for tName in threadList:
        thread = myThread(threadID, tName, workQueue)
        thread.start()
        threads.append(thread)
        threadID += 1

    # Fill the queue
    queueLock.acquire()
    for word in nameList:
        workQueue.put(word)
    queueLock.release()

    # Wait for queue to empty
    while not workQueue.empty():
        pass

    # Notify threads it's time to exit
    exitFlag = 1

    # Wait for all threads to complete
    for t in threads:
        t.join()
    print "Exiting Main Thread"

Upvotes: 1

Views: 1034

Answers (1)

deflator
deflator

Reputation: 169

Turns out, I installed 32-bit python on a 64-bit windows machine accidentally. Strange behavior ensued, mostly from the multiprocessing module. Uninstalling 32-bit and installing 64-bit python 2.7.10 made it work correctly.

So if nothing makes sense with how your code is executing, maybe you've got the wrong interpreter!

Upvotes: 1

Related Questions