CODEWITHSUNDEEP
jsonshellyaml
sorin
sorin

Reputation: 170310

How to update YAML files using shell pipes?

I am looking for a simple UNIX approach for saving data obtained using shell scripts into YAML files, for example I want to save the list of installed packages (rpms, pypi) from the system in a single YAML file using a shell pipes, something like:

rpm -qa | sort | some-yaml-tool manifest.yaml system.packages

The expected results would be a manifest.yaml file that would look like:

system:
    packages:
        - xz-5.2.2-2.fc24.x86_64
        - xz-devel-5.2.2-2.fc24.x86_64

While I prefer YAML, I would not refuse a JSON compatible solution. I am really interested in finding a way to do this with tools that are available on most distributions, not something esoteric that would need to be installed manually or from non-official yum/deb repositories.

If the tool is smart, it should be able to create the file and internal path when it does exist.

Please note that I had to remove jq tool from the list because it does not allow in-place editing.

Upvotes: 1

Views: 1255

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295272

Inserting package names into a YAML file

You don't need in-place editing support for your intended use case at all. Noting that YAML is a superset of JSON, and thus that all output from jq is also valid YAML:

rpm -qa | sort | jq -Rn '{"system": {"packages": [inputs]}}'

will generate an output file of the form:

{
  "system": {
    "packages": [
      "bar",
      "baz",
      "foo"
    ]
  }
}

...in only a single pass. To make this more idiomatic YAML, you can parse through a tiny Python shim using the PyYAML library:

yaml_format() {
  python -c 'import sys, yaml; sys.stdout.write(yaml.dump(yaml.load(sys.stdin), default_flow_style=False))'
}
rpm -qa | sort | jq -Rn '{"system": {"packages": [inputs]}}' | yaml_format

...will generate content of the form:

system:
  packages:
  - bar
  - baz
  - foo

Updating a JSON file in-place

Let's say you already have a JSON file with content other than the manifest that you want to preserve. In that case, the question asked explicitly becomes relevant. A safe (albeit GNU-only) implementation would look like the following:

atomic_update() {
  # usage: atomic_update filename command arg1 arg2 ...
  local filename tempfile retval=0
  filename=$1; shift || return
  tempfile=$(mktemp -t -- "$filename.XXXXXX") || return
  if "$@" <"$filename" >"$tempfile"; then
    # Make a best-effort attempt to preserve permissions
    chown --reference="$filename" -- "$tempfile" &>/dev/null ||:
    chmod --reference="$filename" -- "$tempfile" &>/dev/null ||:
    mv -- "$tempfile" "$filename"
  else
    retval=$?
    rm -f -- "$tempfile"
  fi
  return "$retval"
}

rpm -qa | sort | atomic_update manifest.yml jq -Rn '.system.packages = [inputs]'

Upvotes: 1

sorin
sorin

Reputation: 170310

While I do not have yet fully working solution, I think I am very close to finding one via the json npm packages.

echo "{}" > e.json
seq 3 | xargs -0 -I XXX json -I -f e.json -e 'this.numbers=XXX'
cat e.json

The only part the is missing now is about how to make it use a list and merging to it instead of overriding values.

Upvotes: 0

Related Questions