CODEWITHSUNDEEP
pythonsubprocess
Anton
Anton

Reputation: 115

Subproces.popen - slave write failed: Broken pipe; aborting

TLDR: stuck with this https://code.google.com/archive/p/byte-unixbench/issues/1

Trying to run UnixBench using subprocess.popen() while capturing output and printing it out in realtime.

This is the subroutine I've come up with:

def run_and_print(command, cwd=None, catch_stderr = False):
    if catch_stderr:
        err_pipe = subprocess.PIPE
    else:
        err_pipe = subprocess.STDOUT

    p = subprocess.Popen(command, stdout=subprocess.PIPE, bufsize=1, cwd=cwd, stderr=err_pipe)
    r = ''
    while True:
        if catch_stderr:
            out = p.stderr.read(1)
        else:
            out = p.stdout.read(1)
        if out == "" and p.poll() != None:
            break
        sys.stdout.write(out)
        sys.stdout.flush()
        r += out

    return r

It works just fine for all the purposes except for UnixBench. Unixbench just dies after a while:

unixbench = run_and_print(['./Run'])

...

1 x Pipe Throughput 1 2 3 4 5 6 7 8 9 10

1 x Pipe-based Context Switching 1 2 3 4

Run: "Pipe-based Context Switching": slave write failed: Broken pipe; aborting

Google didn't help much. The only meaningful result I've got is https://code.google.com/archive/p/byte-unixbench/issues/1 and suggest solution to create a java app won't work for me as I need to run the script with as few dependencies as possible.

I'll be thankful for any solution or a workaround. The system I'm testing this on is Ubuntu 14.04.4 x64

Upvotes: 1

Views: 448

Answers (1)

jfs
jfs

Reputation: 414715

The bug is related to 'yes' reporting error with subprocess communicate() which provides the fix: reenable SIGPIPE signal in the child process using preexec_fn (or use Python 3).

Unrelated: your code can deadlock if catch_stderr is true and p.stderr and p.stdout are not perfectly in sync.

Otherwise catch_stderr has no effect (ignoring buffering): your code captures stderr regardless. You could simplify it:

#!/usr/bin/env python
from shutil import copyfileobj
from subprocess import Popen, PIPE, STDOUT

def run_and_print(command, cwd=None):
    p = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=-1, cwd=cwd,
              preexec_fn=restore_signals)
    with p.stdout:
        tee = Tee()
        copyfileobj(p.stdout, tee)        
    return p.wait(), tee.getvalue()

where Tee() is a file-like object that writes to two places: to stdout and to StringIO():

import sys
from io import BytesIO

class Tee:
   def __init__(self):
       self.file = BytesIO()
   def write(self, data):
       getattr(sys.stdout, 'buffer', sys.stdout).write(data)
       self.file.write(data)
   def getvalue(self):
       return self.file.getvalue()

where restore_signals() is defined here.

If you want to see the output on the screen as soon as command prints them; you could inline Tee, copyfileobj() and use os.read(), to avoid reading the complete length before writing it to stdout:

chunks = []
with p.stdout:
    for chunk in iter(lambda: os.read(p.stdout.fileno(), 1 << 13), b''):
        getattr(sys.stdout, 'buffer', sys.stdout).write(chunk)
        sys.stdout.flush()
        chunks.append(chunk)
return p.wait(), b''.join(chunks)

To disable the internal block-buffering in the child process, you might try to run it using stdbuf or pass pseudo-tty instead of the pipe.

Upvotes: 2

Related Questions