CODEWITHSUNDEEP
pythoncommand-linestreamingyaml
Frank Krueger
Frank Krueger

Reputation: 70993

How to process a YAML stream in Python

I have a command line app the continuously outputs YAML data in the form:

- col0: datum0
  col1: datum1
  col2: datum2
- col0: datum0
  col1: datum1
  col2: datum2
...

It does this for all of eternity. I would like to write a Python script that continuously reads each of these records.

The PyYAML library seems best at taking fully loaded strings and interpreting those as a complete YAML document. Is there a way to put PyYAML into a "streaming" mode?

Or is my only option to chunk the data myself and feed it bit by bit into PyYAML?

Upvotes: 6

Views: 3289

Answers (2)

Frank Krueger
Frank Krueger

Reputation: 70993

Here is what I've ended up using since there does not seem to be a built-in method for accomplishing what I want. This function should be generic enough that it can read in a stream of YAML and return top-level objects as they are encountered.

def streamInYAML(stream):
    y = stream.readline()
    cont = 1
    while cont:
        l = stream.readline()
        if len(l) == 0:
            cont = 0
        else:
            if l.startswith(' '):
                y = y + l
            else:
                yield yaml.load(y)
                y = l

Can anyone do better?

Upvotes: 4

John Mulder
John Mulder

Reputation: 10075

All of the references to stream in the the documentation seem to be referring to a stream of documents... I've never tried to use it in the way you describe, but it seems like chunking the data into such a stream of documents is a reasonable approach.

Upvotes: 2

Related Questions