CODEWITHSUNDEEP
pythonsubprocessstdoutpopenros
LPA
LPA

Reputation: 31

Why does Popen.stdout contain only part of output?

I am running two processes simultaneously in python using the subprocess module:

p_topic = subprocess.Popen(['rostopic','echo','/msg/address'], stdout=PIPE)
p_play = subprocess.Popen(['rosbag','play',bagfile_path])

These are ROS processes: p_topic listens for a .bag file to be played and outputs certain information from that .bag file to the stdout stream; I want to then access this output using the p_topic.stdout object (which behaves as a file).

However, what I find happening is that the p_topic.stdout object only contains the first ~1/3 of the output lines it should have - that is, in comparison to running the two commands manually, simultaneously in two shells side by side.

I've tried waiting for many seconds for output to finish, but this doesn't change anything, its approximately the same ratio of lines captured by p_topic.stdout each time. Any hints on what this could be would be greatly appreciated!

EDIT:

Here's the reading code:

#wait for playing to stop
while p_play.poll() == None:
    time.sleep(.1)

time.sleep(X)#wait for some time for the p_topic to finish
p_topic.terminate()

output=[]
for line in p_topic.stdout:
    output.append(line)

Note that the value X in time.sleep(X) doesn't make any difference

Upvotes: 3

Views: 1131

Answers (2)

ShadowRanger
ShadowRanger

Reputation: 155526

By default, when a process's stdout is not connected to a terminal, the output is block buffered. When connected to a terminal, it's line buffered. You expect to get complete lines, but you can't unless rostopic unbuffers or explicitly line buffers its stdout (if it's a C program, you can use setvbuf to make this automatic).

The other (possibly overlapping) possibility is that the pipe buffer itself is filling (pipe buffers are usually fairly small), and because you never drain it, rostopic fills the pipe buffer and then blocks indefinitely until you kill it, leaving only what managed to fit in the pipe to be drained when you read the process's stdout. In that case, you'd need to either spawn a thread to keep the pipe drained from Python, or have your main thread use select module components to monitor and drain the pipe (intermingled with polling the other process). The thread is generally easier, though you do need to be careful to avoid thread safety issues.

Upvotes: 1

Psymon25
Psymon25

Reputation: 356

is it worth trying process communicate/wait? rather than sleep and would that solve your issue?

i have this for general purpose so not sure if you can take this and change it to what you need?

    executable_Params = "{0} {1} {2} {3} {4}".format(my_Binary, 
                                                       arg1, 
                                                       arg2, 
                                                       arg3, 
                                                       arg4)

    # execute the process
    process = subprocess.Popen(shlex.split(executable_Params), 
                               shell=False, 
                               stderr=subprocess.PIPE, 
                               stdout=subprocess.PIPE)


    stdout, stderr = process.communicate()
    ret_code = process.wait()

    if ret_code == 0:
        return 0
    else:
        #get the correct message from my enum method
        error_msg = Process_Error_Codes(ret_code).name
        raise subprocess.CalledProcessError(returncode=ret_code, 
                                            cmd=executable_Params)

Upvotes: 0

Related Questions