CODEWITHSUNDEEP
pythonsubprocessstdin
dariober
dariober

Reputation: 9062

Read stream and pass to subprocess

I'm writing a python script that reads a stream from stdin and passes this stream to subprocess for further processing. The problem is that python hangs after having processed the input stream.

For example, this toy program sorter.py should read from stdin and pass the stream to subprocess for sorting via Unix sort:

cat dat.txt | ./sorter.py

Here's sorter.py:

#!/usr/bin/env python

import subprocess
import sys

p= subprocess.Popen('sort -', stdin= subprocess.PIPE, shell= True)
for line in sys.stdin:
    p.stdin.write(line)
sys.exit()

The stream from cat is correctly sorted but the programs hangs, i.e. sys.exit() is never reached.

I've read quite a few variations on this theme but I can't get it right. Any idea what is missing?

Thank you!

Dario

Upvotes: 3

Views: 1567

Answers (2)

jfs
jfs

Reputation: 414215

My guess: sys.exit() is reached but sort continues to run. You should close p.stdin pipe to signal EOF to sort:

#!/usr/bin/env python2
import subprocess
import sys

p = subprocess.Popen('sort', stdin=subprocess.PIPE, bufsize=-1)
with p.stdin:
    for line in sys.stdin:
        # use line here
        p.stdin.write(line)
if p.wait() != 0:
    raise Error

Example:

$ < dat.txt ./sorter.py

If you don't need to modify the stdin stream then you don't need to use PIPE here:

#!/usr/bin/env python
import subprocess

subprocess.check_call('sort')

Upvotes: 1

DorElias
DorElias

Reputation: 2313

you probably have a problem with buffering -the OS doesnt send the data the moment it arrives to stdin - check this out

https://www.turnkeylinux.org/blog/unix-buffering

Upvotes: 0

Related Questions