CODEWITHSUNDEEP
jsonpipejq
rickhg12hs
rickhg12hs

Reputation: 11902

How to pipe each jq filter output to an external command?

I have a large file with a sequence of JSON objects. Each object has a value that I want to extract and run through an external program. How do you instruct jq to do this?

For example, if I have the following JSON file:

{
  "id": 1,
  "corp": "Slate Rocks",
  "pubKey": "-----BEGIN PUBLIC KEY-----\nMFowDQYJKoZIhvcNAQEBBQADSQAwRgJBAMBUAZGIzQsPO1PXgjdCJUxcZNc6Y4FY\ncVC+JxZIUfaz0WkLI7H4tiKCsd+2F+3Xf+hObSbdoUi1UuT5uzpsda8CAQM=\n-----END PUBLIC KEY-----",
  "update": "2018-12-10"
}
{
  "id": 2,
  "corp": "Pi-ryte",
  "pubKey": "-----BEGIN PUBLIC KEY-----\nMFowDQYJKoZIhvcNAQEBBQADSQAwRgJBALzXcOJvZz2UFPDJphTm++Ho9t+lkTQf\nxH0nSp7lbfCHL5Y5YbnD7pgbD/e/PGIHt+cch3foOlUyH+b0Ht53ZO0CAQM=\n-----END PUBLIC KEY-----",
  "update": "2019-03-14"
}
{
  "id": 3,
  "corp": "Marble Dreams",
  "pubKey": "-----BEGIN PUBLIC KEY-----\nMFowDQYJKoZIhvcNAQEBBQADSQAwRgJBAN1L9v8ZbpJ1/GMlNOfGFjQhhO2KTKpj\nOfp97CDUdCAQ6wzLjagGKbySC16/MpnAvoPUmYEtTRtbdH/rdMGdvd8CAQM=\n-----END PUBLIC KEY-----",
  "update": "2018-12-17"
}

and I want to output the RSA public key moduli, I need to execute openssl for each "pubKey" value.

This only works for the first "pubKey":

$ jq -r '.pubKey' seq.json | openssl rsa -pubin -noout -modulus
Modulus=C054019188CD0B0F3B53D7823742254C5C64D73A6381587150BE27164851F6B3D1690B23B1F8B62282B1DFB617EDD77FE84E6D26DDA148B552E4F9BB3A6C75AF

Is there something like:

$ jq -r '.pubKey | system("openssl rsa -pubin -noout -modulus")' seq.json

or

$ jq -r '.pubKey as $pk | system("openssl rsa -pubin -noout -modulus -in $pk")' seq.json

Upvotes: 1

Views: 2969

Answers (2)

Charles Duffy
Charles Duffy

Reputation: 295272

An implementation that's careful not to execute data as code might look like:

while IFS= read -r -d '' chunk; do
  openssl rsa -pubin -noout -modulus <<<"$chunk"
done < <(jq -j '(.pubKey, "\u0000")' seq.json)
  • The while read idiom is documented in detail in BashFAQ #1
  • The use of process substitution (the <(...) syntax) is for reasons described in BashFAQ #24.
  • The use of the -d argument to read, the -j argument to jq, and the use of "\u0000" inside the JSON uses NUL delimiters to separate extracted data items. These are used because NULs are the only character which can't be represented inside C strings; thus, they can be used to represent delimiters between any data item which a C program can store in a string. See stedolan/jq#1271 for history behind this feature in jq.

Upvotes: 3

glee8e
glee8e

Reputation: 6419

Rather than doing this within jq, as jq cannot use program as filter yet, you can do it outside it:

jq -j --slurp 'map(.pubKey) | join(">")' seq.json | xargs --delimiter=> -n1 --no-run-if-empty -i sh -c 'echo "{}" | openssl rsa -pubin -noout -modulus'

Notes:

  1. Your sample input is multiple object instead of an array of objects, so --slurp is used. If it is not the case you will need to remove this flag.
  2. pubKey cannot have '>' in it, so I used it as an delimiter. This assumes your data comes from trusty source and won't blow your system up.

Upvotes: 0

Related Questions