jq add a field to array in loop and return whole json

Question

I want to add a field to aggregations.domains.buckets array in below json using jq and return the full json with new fields? I want to add {"cost":122.45} where cost is dynamically computed for each key..

{
   "aggregations":{
      "domains":{
         "doc_count_error_upper_bound":0,
         "sum_other_doc_count":0,
         "buckets":[
            {
               "key":"other",
               "doc_count":8679,
               "environments":{
                  "value":49
               }
            },
            {
               "key":"CREATIVESandPREVIEWS",
               "doc_count":1235,
               "environments":{
                  "value":21
               }
            },
            {
               "key":"Integration",
               "doc_count":65,
               "environments":{
                  "value":1
               }
            },
            {
               "key":"devops",
               "doc_count":1,
               "environments":{
                  "value":1
               }
            }
         ]
      }
   }
}

Here is my script:

#!/bin/bash
curl -u xx:xx -H 'Content-Type: application/json' -XPOST 'http://172.21.4.55:9200/log_index-dev-gcecomputeinstances-*/_search?size=0' -o instancesdomainagg.json --data @instancesdomainaggquery.json

var=0
for domain in $(cat instancesdomainagg.json | jq '.aggregations.domains.buckets[].key' | sed -e 's/^"//' -e 's/"$//' | sed -e 's/\(.*\)/\L\1/'); do
curl --trace-ascii curl.trace -u xx:xx -H 'Content-Type: application/json' -XPOST 'http://172.21.4.55:9200/log_index-dev-gcebillingbq-*/_search?size=0' -o bqdomainagg.json --data-binary '{"query":{"bool":{"must":{"term":{"environment_domain.keyword":"'"$domain"'"}},"filter":{"range":{"usage_end_time.value":{"from":"now-1d","to":"now"}}}}},"aggs":{"cost":{"sum":{"field":"cost"}}}}'
val="$(cat bqdomainagg.json | jq '.aggregations.cost.value')"
jq '.aggregations.domains.buckets['"$var"'] |= .+ {"cost": "'"$val"'"}' instancesdomainagg.json > output.json
var=$((var+1))
done

But it's returning json with only last field added. How to dynamically update json?

Thanks in advance.

Answer 1

If the cost is a function of the input JSON, the approach you'd use would be along the lines of the following filter, which merely adds {"cost": null}:

.aggregations.domains.buckets |= map( . + {cost} )

If the cost depends on key according to some other criterion, then it would probably be best to construct a JSON dictionary mapping the keys to the cost, and then pass that dictionary in to jq, and combine the results using the technique illustrated at:

Combining JSON by common key-value pairs

Solution using vals.txt

program.jq:

.aggregations.domains.buckets |=
  reduce range(0; $costs|length) as $i (.;
      .[$i] += {cost: $costs[$i]} )

If you want the cost to be numeric, you'd write:

{cost: $costs[$i]|tonumber } )

Invocation:

jq -f program.jq --argfile costs <(jq -Rs '
  split("\n")|map(select(length>0))' vals.txt) input.json

Output:

{
  "aggregations": {
    "domains": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "other",
          "doc_count": 8679,
          "environments": {
            "value": 49
          },
          "cost": "1280.7631275949711"
        },
        {
          "key": "CREATIVESandPREVIEWS",
          "doc_count": 1235,
          "environments": {
            "value": 21
          },
          "cost": "2210.8813614145324"
        },
        {
          "key": "Integration",
          "doc_count": 65,
          "environments": {
            "value": 1
          },
          "cost": "3143.5814890583424"
        },
        {
          "key": "devops",
          "doc_count": 1,
          "environments": {
            "value": 1
          },
          "cost": "836.4116237582436"
        },
        {
          "cost": "1280.7631275949711"
        }
      ]
    }
  }
}