CODEWITHSUNDEEP
jsonbashshelljq
janneb
janneb

Reputation: 37228

How can I incrementally generate JSON calling jq from bash repeatedly?

Is there some accepted 'best practice' of generating JSON documents using bash and jq? I have a script to gather various data, and to make it easier to further process using other tools I'd like to output the data in JSON format. So I'm using jq to make sure all the quoting etc. gets done correctly, as recommended in this answer: https://stackoverflow.com/a/48470227/75652. However, I'm struggling with how to generate it piecemeal instead of one giant jq call at the end. E.g. something like


read foo <<<$(</path/to/some/oneliner/file)
jq -n --arg f $foo '{foo: $f}'

bar=$(some_command)
jq -n --arg b $bar '{bar: $b}'

Will generate two separate objects (which can be processed with tools that support various more or less informal "JSON streaming" formats, including jq) whereas I'd want a single object, something like


{ "foo": SOMETHING, "bar": SOMETHING_ELSE }

but I can't do that with multiple jq calls as jq will complain that the incomplete JSON is malformed.

And to further add some complexity, in some cases I need to generate nested JSON structures. In another language like python I'd just put all the data in a set of nested dictionaries and then dump it to JSON in the end, but nested dictionaries in bash seem very tedious..

Upvotes: 2

Views: 249

Answers (3)

peak
peak

Reputation: 116870

The Q makes it seem that $foo and $bar can be pre-computed, in which case you can use as a model:

jq -n --arg f "$foo" --arg b "$bar" '.foo = $f | .bar = $b'

Of course if the value of $foo is very large, it would be better to make those values available to jq using a file-oriented command-line option, such as --slurpfile.

If the computation of some of the values depends on very large files, then invoking jq several times might make sense. In that case, making N calls to jq to marshal the values, then making one extra call to assemble them into a single JSON object (perhaps using 'jq -s add') seems very reasonable.

An alternative along the lines suggested in the title of the Q would be to create a pipeline of calls to jq, e.g.:

  jq -n --argfile f <(some nasty stuff) '.foo = $f' |
    jq  --argfile b <(some more nasty stuff) '.bar = $b' | ...

Finally, if $bar depends on $foo in some way, then if that dependence can be expressed in a jq program, you could read in the underlying values in one invocation of jq, using a more complex jq program.

Upvotes: 1

pmf
pmf

Reputation: 36231

When reaching some complexity (or when I need to externally process some of the data between transformations) I typically end up using something along the lines of

jq --slurpfile foo <(
  
  # first inner shell script
  read foo <<<$(</path/to/some/oneliner/file)
  jq -n --arg f $foo '{foo: $f}'

) --slurpfile bar <(

  # second inner shell script
  bar=$(some_command)
  jq -n --arg b $bar '{bar: $b}'

) -n '$foo[0] + $bar[0]'

That way, the outermost jq call may still have a 'real' input on its own, and the inner calls are fairly maintainable with all bash variables in scope.

Upvotes: 1

L&#233;a Gris
L&#233;a Gris

Reputation: 19615

You can save and process intermediary JSON for the next jq command:

#!/usr/bin/env bash

read -r foo <a.txt

json="$(jq -n --arg f "$foo" '{foo: $f}')"


bar="$(pwd)"
jq -n --arg b "$bar" "$json"'+{bar: $b}'

# or alternatively
jq --arg b "$bar" '.bar=$b' <<<"$json"

Upvotes: 1

Related Questions