CODEWITHSUNDEEP
pythonmultithreadingpython-2.7
Wesley
Wesley

Reputation: 3111

Python Multiprocessing Data Exchange

I am trying to pass data between 2 threads within the same process, the main thread spawns a new thread in a separate class. I attempted to pass a string utilizing the multiprocessing.Queue object.

I don't understand why the 2nd thread does not see the data in the queue.

mainClass.py

if(__name__ == '__main__'):
    # Create the threads
    itemInterestThread = NewItemInterestHelper("Item Interest Thread")

    # Start the Item Interest Thread
    itemInterestThread.run()

    time.sleep(1)
    print "Adding item\n"
    itemInterestThread.addToQueue("Neat Thing")
    time.sleep(1)
    print "Adding item\n"
    itemInterestThread.addToQueue("Neat Thing2")

    print "Sleeping for 5 seconds"
    time.sleep(5)

NewItemInterestHelper.py

import multiprocessing
import time

class NewItemInterestHelper:
    # Class Variables
    __itemsQueue = multiprocessing.Queue()
    __itemMutex = multiprocessing.Lock()
    __pool = multiprocessing.Process()

    def __init__(self, name):
        self.name = name

    def run(self):
        self.__pool = multiprocessing.Process(target=self.dowork)
        self.__pool.start()

    def dowork(self):
        while True:
            self.__itemMutex.acquire()
            if not self.__itemsQueue.empty():
                print "Got " + self.__itemsQueue.get() + "\n"

            self.__itemMutex.release()
            time.sleep(1)

    def addToQueue(self, queueAddition):
        self.__itemMutex.acquire()
        self.__itemsQueue.put(queueAddition)
        self.__itemMutex.release()

Upvotes: 0

Views: 862

Answers (1)

JohanL
JohanL

Reputation: 6891

First of all, you are confusing threading and multiprocessing in your question. They are not the same thing. It would probably be good if you update your question to be clearer. Threading is done in the same process (conceptually at least) and will therefore share all global variables and states. When doing multiprocessing, the processes will be independent of each other and will therefore not share the global variables and states. Instead, each process will instantiate their own.

Because of this, your class variables (which in effect are global variables) will not be shared between your main and your auxiliary process. The different processes will have their own queues and locks, created from scratch.

However, if the lock and queue instead belong to the object, rather than the class, they will be copied when the auxiliary process is spawned. That way they will share enough of their internals to do what you want them to do. Thus if you change your NewItemInterestHelper to make the lock and the queue being part of the dynamic object instead, it should work:

class NewItemInterestHelper:
    # Removing all class variables

    def __init__(self, name):
        self.__itemsQueue = multiprocessing.Queue()
        self.__itemMutex = multiprocessing.Lock()
        # removing self.__pool since it is just overwritten
        # by the run method

        self.name = name

    # Rest of class follows...

Now, since the queue and the lock in the auxiliary process are instead created from a copy of the queue and the lock in the main process. That way they will share enough of their internals to be considered the same queue and lock respectively.

On another note, I assume this to be a simplified example, because as written, there is no need for the mutex/lock at all. There is no need to protect the writing to the queue from the reading from the queue, as that is automatically handled by the underlying system.

Upvotes: 1

Related Questions