CODEWITHSUNDEEP
pythonsubprocess
chrislamp
chrislamp

Reputation: 127

How can I execute a shell script from stdin and get the output in realtime using python?

I want to mimic the below using python subprocess:

cat /tmp/myscript.sh | sh

The /tmp/myscript.sh contains:

ls -l
sleep 5
pwd

Behaviour: stdout shows the result of "ls" and the results of "pwd" are shown after 5 seconds.

What I have done is:

import subprocess      
f = open("/tmp/myscript.sh", "rb")
p = subprocess.Popen("sh", shell=True, stdin=f, 
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
f.close()
p.stdout.read()

This waits until ALL the processing is done and shows the results all at once. The desired effect is to fill in the stdout pipe in realtime.

Note: This expectation seems non sense but this is sample from a bigger and complex situation which I cannot describe here.

Another Note: I can't use p.communicate. This whole thing is inside a select.select statement so I need stdout to be in a pipe.

Upvotes: 0

Views: 456

Answers (1)

Barmar
Barmar

Reputation: 780851

The problem is that when you don't give an argument to read(), it reads until EOF, which means it has to wait until the subprocess exits and the pipe is closed.

If you call it with a small argument it will return immediately after it has read that many characters

import subprocess      
f = open("/tmp/myscript.sh", "rb")
p = subprocess.Popen("sh", shell=True, stdin=f, 
        stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='utf-8')
f.close()
while True:
    c = p.stdout.read(1)
    if not c:
        break
    print(c, end='')
print()

Note that some many buffer their output when stdout is connected to a pipe, so this might not solve the problem for everything. The shell doesn't buffer its own output, but ls probably does. But since ls is producing all its output at once, it won't be a problem in this case.

To solve the more general problem you may need to use a pty instead of a pipe. The pexpect library is useful for this.

Upvotes: 1

Related Questions