CODEWITHSUNDEEP
jsonbashcountjqmissing-data
iskandarblue
iskandarblue

Reputation: 7526

Count records with missing keys using jq

Below is a sample output that is returned when calling an API:

curl "https://mywebsite.com/api/cars.json&page=1" |  jq '.'

Using jq, how would one count the number or records where the charge key is missing? I understand that the first bit of code would include jq '. | length' but how would one filter out objects that contain or don't contain a certain key value ?

If applied to the sample below, the output would be 1

{
  "current_page": 1,
  "items": [
    {
      "id": 1,
      "name": "vehicleA",
      "state": "available",
      "charge": 100
    },
    {
      "id": 2,
      "name": "vehicleB",
      "state": "available",
    },
    {
      "id": 3,
      "name": "vehicleB",
      "state": "available",
      "charge": 50
    }
  ]
}

Upvotes: 8

Views: 7845

Answers (2)

peak
peak

Reputation: 116919

Here is a solution that uses a simple but powerful utility function worthy perhaps of your standard library:

def sigma(stream): reduce stream as $s (null; . + $s);

The filter you'd use with this would be:

sigma(.items[] | select(has("charge") == false) | 1)

This is very efficient as no intermediate array is required, and no useless additions of 0 are involved. Also, as mentioned elsewhere, using has is more robust than making assumptions about the value of .charge.

Startup file

If you have no plans to use jq's module system, you can simply add the above definition of sigma to the file ~/.jq and invoke jq like so:

jq 'sigma(.items[] | select(has("charge") == false) | 1)'

Better yet, if you also add def count(s): sigma(s|1); to the file, the invocation would simply be:

jq 'count(.items[] | select(has("charge") | not))'

Standard Library

If for example ~/.jq/jq/jq.jq is your standard library, then assuming count/1 is included in this file, you could invoke jq like so:

jq 'include "jq"; count(.items[] | select(has("charge") == false))'

Upvotes: 3

jq170727
jq170727

Reputation: 14715

Here is a solution using map and length:

.items | map(select(.charge == null)) | length

Try it online at jqplay.org

Here is a more efficient solution using reduce:

reduce (.items[] | select(.charge == null)) as $i (0;.+=1)

Try it online at jqplay.org

Sample Run (assuming corrected JSON data in data.json)

$ jq -M 'reduce (.items[] | select(.charge == null)) as $i (0;.+=1)' data.json
1

Note that each of the above takes a minor shortcut assuming that the items won't have a "charge":null member. If some items could have a null charge then the test for == null won't distinguish between those items and items without the charge key. If this is a concern the following forms of the above filters which use has are better:

.items | map(select(has("charge")|not)) | length

reduce (.items[] | select(has("charge")|not)) as $i (0;.+=1)

Upvotes: 17

Related Questions