Remove all null values

Question

I am trying to remove null values from a json object using jq. I found this issue on their github and so now I'm trying to remove them with del. I have this:

'{ id: $customerId, name, phones: ([{ original: .phone }, 
 { original: .otherPhone}]), email} | del(. | nulls)'

This doesn't seem to do anything. However if I replace nulls with .phones it does remove the phone numbers.

Answer 1

The following illustrates how to remove all the null-valued keys from a JSON object:

jq -n '{"a":1, "b": null, "c": null} | with_entries( select( .value != null ) )'
{
  "a": 1
}

Alternatively, paths/0 can be used as follows:

. as $o | [paths[] | {(.) : ($o[.])} ] | add

By the way, del/1 can also be used to achieve the same result, e.g. using this filter:

reduce keys[] as $k (.; if .[$k] == null then del(.[$k]) else . end)

Or less obviously, but more succinctly:

del( .[ (keys - [paths[]])[] ] )

And for the record, here are two ways to use delpaths/1:

jq -n '{"a":1, "b": null, "c": null, "d":2} as $o
  | $o
  | delpaths( [  keys[] | select( $o[.] == null ) ] | map( [.]) )'


$ jq -n '{"a":1, "b": null, "c": null, "d":2}
  | [delpaths((keys - paths) | map([.])) ] | add'

In both these last two cases, the output is the same: { "a": 1, "d": 2 }

---

For reference, if you wanted to remove null-valued keys from all JSON objects in a JSON text (i.e., recursively), you could use walk/1, or:

del(.. | objects | (to_entries[] | select(.value==null) | .key) as $k | .[$k])

Answer 2

This answer by Michael Homer on https://unix.stackexchange.com has a super concinse solution which works since jq 1.6:

del(..|nulls)

It deletes all null-valued properties (and values) from your JSON. Simple and sweet :)

nulls is a builtin filter and can be replaced by custom selects:

del(..|select(. == "value to delete"))

To remove elements based on multiple conditions, e.g. remove all bools and all numbers:

del(..|booleans,numbers)

or, to only delete nodes not matching a condition:

del(..|select(. == "value to keep" | not))

(The last example is only illustrative – of course you could swap == for !=, but sometimes this is not possible. e.g. to keep all truthy values: del(..|select(.|not)))

Answer 3

With newer versions of jq (1.6 and later)

You can use this expression to remove null-valued keys recursively:

jq 'walk( if type == "object" then with_entries(select(.value != null)) else . end)'

REF

Answer 4

That's not what del/1 was meant to be used for. Given an object as input, if you wanted to remove the .phones property, you'd do:

del(.phones)

In other words, the parameter to del is the path to the property you wish to remove.

If you wanted to use this, you would have to figure out all the paths to null values and pass it in to this. That would be more of a hassle though.

---

Streaming the input in could make this task even simpler.

fromstream(tostream | select(length == 1 or .[1] != null))

---

Otherwise for a more straightforward approach, you'll have to walk through the object tree to find null values. If found, filter it out. Using walk/1, your filter could be applied recursively to exclude the null values.

walk(
    (objects | with_entries(select(.value != null)))
    // (arrays | map(select(. != null)))
    // values
)

So if you had this for input:

{
    "foo": null,
    "bar": "bar",
    "biz": [1,2,3,4,null],
    "baz": {
        "a": 1,
        "b": null,
        "c": ["a","b","c","null",32,null]
    }
}

This filter would yield:

{
    "bar": "bar",
    "baz": {
        "a": 1,
        "c": ["a","b","c","null",32]
    },
    "biz": [1,2,3,4]
}

Answer 5

All the other answers to date here are workarounds for old versions of jq, and it isn't clear how do do this simply in the latest released version. In JQ 1.6 or newer this will do the job to remove nulls recursively:

$ jq 'walk( if type == "object" then with_entries(select(.value != null)) else . end)' input.json

Sourced from this comment on the issue where adding the walk() function was discussed upstream.

Answer 6

Elsewhere on this page, some interest has been expressed in using jq to eliminate recursively occurrences of [] and {} as well as null.

Although it is possible to use the built-in definition of walk/1 to do this, it is a bit tricky to do so correctly. Here therefore is a variant version of walk/1 which makes it trivial to do so:

def traverse(f):
  if type == "object" then map_values(traverse(f)) | f
  elif type == "array" then map( traverse(f) ) | f
  else f
  end;

In order to make it easy to modify the criterion for removing elements, we define:

def isempty: .==null or ((type|(.=="array" or .=="object")) and length==0);

The solution is now simply:

traverse(select(isempty|not))

Answer 7

[WARNING: the definition of walk/1 given in this response is problematic, not least for the reason given in the first comment; note also that jq 1.6 defines walk/1 differently.]

I am adding the new answer to emphasize the extended version of the script by @jeff-mercado. My version of the script assumes the empty values are as follows:

• null;
• [] - empty arrays;
• {} - empty objects.

Removing of empty arrays and objects was borrowed from here https://stackoverflow.com/a/26196653/3627676.

def walk(f):
    . as $in | 
    if type == "object" then
        reduce keys[] as $key
            ( {}; . + { ($key): ( $in[$key] | walk(f) ) } ) | f
    elif type == "array" then 
        select(length > 0) | map( walk(f) ) | f
    else 
        f
    end;

walk(
    if type == "object" then
        with_entries(select( .value != null and .value != {} and .value != [] ))
    elif type == "array" then
        map(select( . != null and . != {} and .!= [] ))
    else
        .
    end
)

Answer 8

[WARNING: the response below has several problems, not least those arising from the fact that 0|length is 0.]

Elaborating on an earlier answer, in addition to removing properties with null values, it's often helpful to remove JSON properties with empty array or object values (i.e., [] or {}), too.

ℹ️ — jq's walk() function (walk/1) makes this easy. walk() will be available in a future version of jq (> jq-1.5), but its definition can be added to current filters.

The condition to pass to walk() for removing nulls and empty structures is:

walk(
  if type == "object" then with_entries(select(.value|length > 0))
  elif type == "array" then map(select(length > 0))
  else .
  end
)

Given this JSON input:

{
  "notNullA": "notNullA",
  "nullA": null,
  "objectA": {
    "notNullB": "notNullB",
    "nullB": null,
    "objectB": {
      "notNullC": "notNullC",
      "nullC": null
    },
    "emptyObjectB": {},
    "arrayB": [
      "b"
    ],
    "emptyBrrayB": []
  },
  "emptyObjectA": {},
  "arrayA": [
    "a"
  ],
  "emptyArrayA": []
}

Using this function gives the result:

{
  "notNullA": "notNullA",
  "objectA": {
    "notNullB": "notNullB",
    "objectB": {
      "notNullC": "notNullC"
    },
    "arrayB": [
      "b"
    ]
  },
  "arrayA": [
    "a"
  ]
}