CODEWITHSUNDEEP
pythonlinuxpipe
chew socks
chew socks

Reputation: 1446

How to tell when data through a pipe has ended

In my python program I have two subprocesses interconnected by a pipe and with one connected to stdin and the other connected to stdout. My problem is that when the data flow ends the subprocesses hang until I press ctrl+c. It looks to me like the subprocesses are being held open my the pipe. If I could tell when the data flowing through the pipe I could close it manually.

def write(tag_name):
    p_r, p_w = os.pipe()
    pv = subprocess.Popen('pv', stdin=None, stdout=p_w)
    dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=p_r, stdout=None)
    dd.wait()

Upvotes: 1

Views: 1274

Answers (2)

Jean-François Fabre
Jean-François Fabre

Reputation: 140256

Just don't use os.pipe(), you can pass subprocess stdout directly to the other process stdin, like this:

def write(tag_name):
    pv = subprocess.Popen('pv', stdin=None, stdout=subprocess.PIPE)
    dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=pv.stdout, stdout=None)
    dd.wait()

When first command ends, the pipe is broken (as opposed to os.pipe() which need to be closed manually), so it ends the second command as well and the script can continue/end.

I have tested a simple pipe command and with os.pipe() it blocks at the end as you described, but exited when first process ended with my modifications.

Upvotes: 2

Veselin Penev
Veselin Penev

Reputation: 77

You need a non-blocking solution here. Take a look at my solution: https://github.com/vesellov/bitdust.devel/blob/master/system/nonblocking.py

And you can call it this way (did not tested the code):

import nonblocking, time
p = nonblocking.Popen('pv'.split(), shell=True, )
p.make_nonblocking()
while 1:
    if p.state() == nonblocking.PIPE_CLOSED:
        # pipe closed, stop
        return break
    if p.state() == nonblocking.PIPE_READY2READ:
        newchunk = p.recv(1024)
        if newchunk == '':
            # EOF reached, stop
            break
        # do something with the data here
        # you can send it to second stream
    try:
       time.sleep(0.01)
    except KeyboardInterrup:
       break

So when you call dd.wait() it will block, that is why your Ctrl-C not working. You need to deal with this manually.... non-blocking streaming is not a trivial story in Python. Check-out Twisted project, you can find a lot of cool stuff :-)

Upvotes: 0

Related Questions