CODEWITHSUNDEEP
pythonsubprocess
tbeauvais
tbeauvais

Reputation: 1738

Python subprocess timing out?

I have a script that runs another command, waits for it to finish, logs the stdout and stderr and based the return code does other stuff. Here is the code:

p = subprocess.Popen(command, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
o, e = p.communicate()
if p.returncode:
    # report error

# do other stuff

The problem I'm having is that if command takes a long time to run none of the other actions get done. The possible errors won't get reported and the other stuff that needs to happen if no errors doesn't get done. It essentially doesn't go past p.communicate() if it takes too long. Some times this command can takes hours (or even longer) to run and some times it can take as little as 5 seconds.

Am I missing something or doing something wrong?

Upvotes: 0

Views: 208

Answers (2)

Antti Haapala -- Слава Україні
Antti Haapala -- Слава Україні

Reputation: 133879

The Popen.communicate waits for the process to finish, before anything is returned. Thus it is not ideal for any long running command; and even less so if the subprocess can hang waiting for input, say prompting for a password.

The stderr=subprocess.PIPE, stdout=subprocess.PIPE are needed only if you want to capture the output of the command into a variable. If you are OK with the output going to your terminal, then you can remove these both; and even use subprocess.call instead of Popen. Also, if you do not provide input to your subprocess, then do not use stdin=subprocess.PIPE at all, but direct that from the null device instead (in Python 3.3+ you can use stdin=subprocess.DEVNULL; in Python <3.3 use stdin=open(os.devnull, 'rb')

If you need the contents too, then instead of calling p.communicate(), you can read p.stdout and p.stderr yourself in chunks and output to the terminal, but it is a bit complicated, as it is easy to deadlock the program - the dummy approach would try to read from the subprocess' stdout while the subprocess would want to write to stderr. For this case there are 2 remedies:

  • you could use select.select to poll both stdout and stderr to see whichever becomes ready first and read from it then

  • or, if you do not care for stdout and stderr being combined into one, you can use STDOUT to redirect the stderr stream into the stdout stream: stdout=subprocess.PIPE, stderr=subprocess.STDOUT; now all the output comes to p.stdout that you can read easily in loop and output the chunks, without worrying about deadlocks:

If the stdout, stderr are going to be huge, you can also spool them to a file right there in Popen; say,

stdout = open('stdout.txt', 'w+b')
stderr = open('stderr.txt', 'w+b')

p = subprocess.Popen(..., stdout=stdout, stderr=stderr)
while p.poll() is None:
    # reading at the end of the file will return an empty string
    err = stderr.read()  
    print(err)
    out = stdout.read()
    print(out)
    # if we met the end of the file, then we can sleep a bit
    # here to avoid spending excess CPU cycles just to poll;
    # another option would be to use `select`
    if not err and not out:  # no input, sleep a bit
        time.sleep(0.01)

Upvotes: 0

user3715319
user3715319

Reputation: 31

As per the documentation located here, it's safe to say that you're code is waiting for the subprocess to finish.

If you need to go do 'other things' while you wait you could create a loop like:

while p.poll():
    # 'other things'
    time.sleep(0.2)

Pick a sleep time that's reasonable for how often you want python to wake up and check the subprocess as well as doing its 'other things'.

Upvotes: 1

Related Questions