CODEWITHSUNDEEP
pythonsubprocesspopen
MaxTheMooshroom
MaxTheMooshroom

Reputation: 71

How can I stream input into a python Popen subprocess and stream output from it?

I'm attempting to basically create a shell in python using subprocess.Popen. As boilerplate code for testing purposes, this is what I have:

if __name__ == '__main__':
    ps = subprocess.Popen(
        input('command? '),
        shell=True,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True
    )

    print(ps.stdout.read())
    time.sleep(1) # provide enough time to register completion if the command is a one-off, like dir or echo
    while ps.poll() == None: # ps.poll() returns None if still running
        try:
            # method one
            ps.stdin.write(input())
            ps.communicate()
            print(ps.stdout.read())

            # method two
            ps.stdin.write(input())
            print(ps.communicate()[0])

            # method three
            print(ps.communicate(input())[0])
        except:
            continue

    print('Process Finished')

Each method is a different (failed) attempt.

For commands like python which should open the python CLI interpreter, this completely fails. However, for one-off commands like dir or ls or even running python -c "<some python>" it works just fine.

CLI log:

C:\Users\nj2u2\Desktop\test>python test.py
command? echo hello
hello

Process Finished

C:\Users\nj2u2\Desktop\test>python test.py
command? dir
 Volume in drive C has no label.
 Volume Serial Number is D6B7-6B8D

 Directory of C:\Users\nj2u2\Desktop\test

07/03/2020  12:26 AM    <DIR>          .
07/03/2020  12:26 AM    <DIR>          ..
07/03/2020  08:20 PM             6,811 subprocess_plus.py
07/04/2020  12:55 PM               580 test.py
07/03/2020  08:25 PM    <DIR>          __pycache__
               2 File(s)          7,391 bytes
               3 Dir(s)  1,487,446,302,720 bytes free

Process Finished

C:\Users\nj2u2\Desktop\test>python test.py
command? python

After that last command, python, it just hangs at print(ps.stdout.read()).

I'd like to know why it's hanging, and how I can fix it.

Upvotes: 2

Views: 1160

Answers (1)

Serge Ballesta
Serge Ballesta

Reputation: 149175

The communicate method can only be used for non interactive commands because it sends a one shot input and then waits for the sub process to terminate. That is the reason that it can only work with what you call one-off commands.

But beware, pipes do not support the terminal discipline in which a read call automatically flushes the output, and where a new line causes a read to return immediately. So nothing guarantees that you can use that to provide interactive IO to the python CLI interpretor. Pipes are just streams, and can be buffered so you can just be sure that all the input will be delivered, and that when the output channels will be closed (i.e. when the sub-process command will terminate) you will receive all the output. The way input and output are intermixed depends on the way the sub-process command is programmed. If it willingly flushes its output before reading, and you do the same in your code, then interactive processing is possible, but if it does not, there is no way to expect to receive something after each end of line.

Upvotes: 1

Related Questions