CODEWITHSUNDEEP
python-3.xsubprocessbuffering
michjnich
michjnich

Reputation: 3395

Output data from subprocess command line by line

I am trying to read a large data file (= millions of rows, in a very specific format) using a pre-built (in C) routine. I want to then yeild the results of this, line by line, via a generator function.

I can read the file OK, but where as just running:

<command> <filename>

directly in linux will print the results line by line as it finds them, I've had no luck trying to replicate this within my generator function. It seems to output the entire lot as a single string that I need to split on newline, and of course then everything needs reading before I can yield line 1.

This code will read the file, no problem: 

import subprocess
import config

file_cmd = '<command> <filename>'

for rec in (subprocess.check_output([file_cmd], shell=True).decode(config.ENCODING).split('\n')):
    yield rec

(ENCODING is set in config.py to iso-8859-1 - it's a Swedish site)

The code I have works, in that it gives me the data, but in doing so, it tries to hold the whole lot in memory. I have larger files than this to process which are likely to blow the available memory, so this isn't an option.

I've played around with bufsize on Popen, but not had any success (and also, I can't decode or split after the Popen, though I guess the fact I need to split right now is actually my problem!).

Upvotes: 3

Views: 815

Answers (1)

michjnich
michjnich

Reputation: 3395

I think I have this working now, so will answer my own question in the event somebody else is looking for this later ... 

proc = subprocess.Popen(shlex.split(file_cmd), stdout=subprocess.PIPE)
while True:
    output = proc.stdout.readline()

    if output == b'' and proc.poll() is not None:
        break
    if output:
        yield output.decode(config.ENCODING).strip()

Upvotes: 5

Related Questions