CODEWITHSUNDEEP
pythonpopen
sashoalm
sashoalm

Reputation: 79467

Avoid subprocess.Popen's stdout pipe clogging and freezing the child process

I have a use case when I need to start a process (IIS Express web server), wait until it prints a certain line to be certain it's fully initialized, and then stop reading its output and run some other blocking operation:

proc = subprocess.Popen('iisexpress.exe /path:<bla bla bla>', shell=True, stdout=subprocess.PIPE);
# .... <read some output>
os.system('nunit <bla bla bla>') # This takes a long time

The problem is that I stop reading the output pipe but the child process doesn't stop writing to it. Eventually the pipe fills up and the child process freezes when trying to write to it.

I am thinking about 2 approaches - either start a new thread in my Python script to read and discard lines from the child process, or find some way to redirect the pipe to /dev/null after I've finished reading from it.

Which of the approaches should I take?

Upvotes: 1

Views: 567

Answers (1)

Alfe
Alfe

Reputation: 59426

You will have to read all the stuff from the first process. You might do this by forking yourself and reading (and forgetting) everything in the child process or by splitting off a separate thread to do it. I would prefer the thread approach:

import threading

proc = subprocess.Popen('iisexpress.exe /path:<bla bla bla>',
                        shell=True, stdout=subprocess.PIPE)

# now wait for the trigger line:
while trigger not in proc.stdout.readline():
    pass

# start a thread for reading the rest:
def read_and_drop():
    while proc.stdout.readline():
        pass

threading.Thread(target=read_and_drop)

# here start the other process:
os.system('nunit <bla bla bla>')

I assume that the first process will print its output in a line-by-line fashion. If instead it prints out 4TB of text without newlines, this will not work properly (because the thread tries to read complete lines). Please comment if this is an issue for you, then I can give a slightly less readable and slightly longer version which deals with this as well.

Upvotes: 2

Related Questions