CODEWITHSUNDEEP
bash
Miles
Miles

Reputation: 1635

Replace string in YAML with stringified JSON

Say you have a template YAML file:

foo:
  bar: {%VARIABLE%}

and want the value of bar to be a stringified JSON? Ex:

foo:
  bar: '{ "hey": "there" }'

In my case, the JSON is in a file, so I'm doing this:

VAR=$(cat my.json)
sed -i '' "s|{%VARIABLE%}|'$VAR'|g" foo.yaml

but if you leave the newlines in the JSON, sed complains.

If I use cat my.json | tr -d '\n' to remove the newlines, the program ingesting my JSON says the private key in the JSON is invalid.

What's the proper way to do this?

Upvotes: 1

Views: 1216

Answers (2)

Anthon
Anthon

Reputation: 76578

What you have there is not a YAML file, or even a template YAML file, as that is not valid YAML: the (flow) mapping start indicator { is not followed by a key-value pair.

What you do have is a template for a YAML file, which is using the jinja2 templating, and there is plug-in for ruamel.yaml available to update exactly that kind of templates (in fact usually a bit more complex than yours). That plugin converts the template to a valid YAML file, which can then be loaded, updated and dumped safely using Python.

This has the major advantage in that the value node can be set to be represented as a literal style scalar and that there is no problem with whatever JSON you throw at it, as your stringified JSON would at least need escaping of quotes (and possible of backslashes when using double quotes).

import sys
import ruamel.yaml

yaml_str = """\
foo:
  bar: {%VARIABLE%}
"""

json_str = """\
{
  "hey": "there's cake"
}
"""

yaml = ruamel.yaml.YAML(typ='jinja2')
data = yaml.load(yaml_str)

data['foo']['bar'] = ruamel.yaml.scalarstring.LiteralScalarString(json_str)
del data['foo'].ca._items['bar']  

yaml.dump(data, sys.stdout)

which gives:

foo:
  bar: |
    {
      "hey": "there's cake"
    }

Normally the non-jinja2 part of the template is edited as YAML, in your case the del is needed, otherwise the %VARIABLE% would show up in a comment in the ouput.

I on purpose changed the value in the JSON to include a single quote. To include that in your example output you would need to do:

foo:
  bar: '{ "hey": "there''s cake" }'

While this might still be doable using non-YAML aware tools like sed, taking care of multi-line JSON with empty lines properly is no trivial. All these problems go away by inserting the element as a literal style scalar, but this has to be done using a proper parser, as that can e.g. take proper action when encountering a first line in your JSON that is less indented than your second, etc.

Upvotes: 1

ErikMD
ErikMD

Reputation: 14723

I'd suggest using perl instead, which is more robust than sed, and allows writing some one-liners as well, such as:

VAR=$(<my.json)
perl -i.bak -wpe "s|{%VARIABLE%}|'$VAR'|g" foo.yaml

(tested with the file my.json below)

{ "hey" : "there",
  "status" : "ok" }

Finally, note that if the .json file contains characters such as $ and @ that could have a special meaning in the replacement string, it would be worth it to rely on Perl's quoting operator q{}:

perl -i.bak -wpe "my \$x=q{$VAR}; s|{%VARIABLE%}|'\$x'|g" foo.yaml

or alternatively:

perl -i.bak -wpe "my \$x='$VAR'; s|{%VARIABLE%}|'\$x'|g" foo.yaml

Upvotes: 1

Related Questions