CODEWITHSUNDEEP
python
Antoine Cotten
Antoine Cotten

Reputation: 2762

How to properly redirect stdin to multiple subprocesses created sequentially?

Context

I am experimenting with a script that is similar to vegeta's ramp-requests.py. In this script, I am running multiple subprocesses sequentially using subprocess.run(), and expect the standard input of the script to be redirected to those subprocesses during their entire lifetime (5s each).

#!/usr/bin/env python3

import json
import os
import subprocess
import sys
import time

rates = [1.0, 2.0, 3.0, 4.0]

# Run vegeta attack
for rate in rates:
    filename='results_%i.bin' % (1000*rate)
    if not os.path.exists(filename):
        cmd = 'vegeta attack -format=json -lazy --duration 5s -rate %i/1000s -output %s' % (1000*rate, filename)
        print(cmd, file=sys.stderr)
        subprocess.run(cmd, shell=True, encoding='utf-8')

I invoke the script as follows, by piping an infinite amount of inputs to it, each input separated by a new line. vegeta reads this input continuously until --duration has elapsed:

$ target-generator | ./ramp-requests.py

Problem

The first subprocess (rate=1.0) seems to receive stdin as I expect, and the command runs successfully, every time.

The second iteration (rate=2.0), however, fails silently, along with all subsequent iterations. If I inspect the corresponding report files (e.g. results_2000.bin) using the vegeta report command, I see fragments of errors such as parse error: syntax error near offset 0 of 'ource":["c...'.

My intuition is telling me that the second subprocess started consuming the input where the first one left it, in the middle of a line, but injecting a sys.stdin.readline() after subprocess.run() doesn't solve it. If that is the case, how can I cleanly solve this issue and ensure each subprocess starts reading from a "good" position?

Upvotes: 0

Views: 203

Answers (2)

Antoine Cotten
Antoine Cotten

Reputation: 2762

As mentioned in @Barmar's comments, Python 3 opens stdin in buffered text mode, so both sys.stdin.read(1) and sys.stdin.readline() cause a read ahead and do not reposition the sys.stdin stream to the beginning of a new line.

There is, however, a way to disable buffering by opening sys.stdin in binary mode, as pointed out by Denilson Sá Maia in his answer to Setting smaller buffer size for sys.stdin?:

unbuffered_stdin = os.fdopen(sys.stdin.fileno(), 'rb', buffering=0)

By doing so, it is possible to read the truncated input until the end of the line from this unbuffered io object after each subprocess returns:

# Run vegeta attack
for rate in rates:
  # [...]

  cmd = 'vegeta attack [...]'
  subprocess.run(cmd, shell=True, encoding='utf-8')

  # Read potentially truncated input until the next '\n' byte
  # to reposition stdin to a location that is safe to consume.
  unbuffered_stdin.readline()

Printing the read line shows something similar to the output below:

b'a4b-b142-fabe0e96a6ca"],"Ce-Type":["perf.drill"],"Ce-Source":["load-test"]}}\n'

All subprocesses are now being executed successfully:

$ for r in results_*.bin; do vegeta report "$r"; done
[...]
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:5
Error Set:
[...]
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:7
Error Set:
[...]
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:8
Error Set:
[...]

See also io - Raw I/O (Python 3 docs)

Upvotes: 0

dstromberg
dstromberg

Reputation: 7177

Read a number of lines from stdin in your parent process, and pass that to your child process as -its- stdin. Repeat as needed. In this way, you do not need to worry about a child process making a mess of your stdin.

Feel free to borrow ideas from https://stromberg.dnsalias.org/~strombrg/mtee.html

HTH

Upvotes: 1

Related Questions