CODEWITHSUNDEEP
pythonmodulequeuemultiprocessing
rwest
rwest

Reputation: 21

Trying to understand multiprocessing and queues across python modules

I'm trying to understand multiprocessing. My actual application is to display log messages in real time on a pyqt5 GUI, but I ran into some problems using queues so I made a simple program to test it out.

The issue I'm seeing is that I am unable to add elements to a Queue across python modules and across processes. Here is my code and my output, along with the expected output.

Config file for globals:

# cfg.py
# Using a config file to import my globals across modules

#import queue
import multiprocessing

# q = queue.Queue()
q = multiprocessing.Queue()

Main module:

# mod1.py

import cfg
import mod2
import multiprocessing

def testq():
    global q
    print("q has {} elements".format(cfg.q.qsize()))

if __name__ == '__main__':
    testq()
    p = multiprocessing.Process(target=mod2.add_to_q)
    p.start()
    p.join()
    testq()
    mod2.pullfromq()
    testq()

Secondary module:

# mod2.py

import cfg

def add_to_q():
    cfg.q.put("Hello")
    cfg.q.put("World!")
    print("qsize in add_to_q is {}".format(cfg.q.qsize()))


def pullfromq():
    if not cfg.q.empty():
        msg = cfg.q.get()
        print(msg)

Here is the output that I actually get from this:

q has 0 elements
qsize in add_to_q is 2
q has 0 elements
q has 0 elements

vs the output that I would expect to get:

q has 0 elements
qsize in add_to_q is 2
q has 2 elements
Hello
q has 1 elements

So far I have tried using both multiprocessing.Queue and queue.Queue. I have also tested this with and without Process.join().

If I run the same program without using multiprocessing, I get the expected output shown above.

What am I doing wrong here?

EDIT:

Process.run() gives me the expected output, but it also blocks the main process while it is running, which is not what I want to do.

My understanding is that Process.run() runs the created process in the context of the calling process (in my case the main process), meaning that it is no different from the main process calling the same function.

I still don't understand why my queue behavior isn't working as expected

Upvotes: 1

Views: 150

Answers (1)

rwest
rwest

Reputation: 21

I've discovered the root of the issue and I'll document it here for future searches, but I'd still like to know if there's a standard solution to creating a global queue between modules so I'll accept any other answers/comments.

I found the problem when I added the following to my cfg.py file.

print("cfg.py is running in process {}".format(multiprocessing.current_process()))

This gave me the following output:

cfg.py is running in process <_MainProcess(MainProcess, started)>
cfg.py is running in process <_MainProcess(Process-1, started)>
cfg.py is running in process <_MainProcess(Process-2, started)>

It would appear that I'm creating separate Queue objects for each process that I create, which would certainly explain why they aren't interacting as expected.

This question has a comment stating that

a shared queue needs to originate from the master process, which is then passed to all of its subprocesses.

All this being said, I'd still like to know if there is an effective way to share a global queue between modules without having to pass it between methods.

Upvotes: 1

Related Questions