Unexpected null elements in output array

Question

EDIT: Now after 16 hours break I realized I have been misreading the results for hours - I didn't notice that there is null in all cases, so the behavior is consistent unlike I claim in this question (facepalm). However I decided not to delete this question but only close it as there is already useful answers.

Consider the following input JSON:

{
  "widgets": [
    {
      "type": "FOO",
      "id": "F1"
    },
    {
      "type": "ZAP",
      "id": "Z1"
    },
    {
      "type": "BAR",
      "id": "B1"
    }
  ]
}

The following transformation:

[
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "type": {
            "FOO": {
              "@(2,id)": "widgets[&3].fooId"
            },
            "BAR": {
              "@(2,id)": "widgets[&3].barId"
            }
          }
        }
      }
    }
  }
]

creates the expected (correct) output. EDIT I misread this output - I didn't realize there is 3 elements where one element is null but I though there is only 2 non-null elements:

{
  "widgets": [
    {
      "fooId": "F1"
    },
    null,
    {
      "barId": "B1"
    }
  ]
}

However the following transformation creates the unexpected output:

[
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "type": {
            "BAR": {
              "@(2,id)": "widgets[&3].barId"
            }
          }
        }
      }
    }
  }
]

This is the actual (wrong) output:

{
  "widgets": [
    null,
    null,
    {
      "barId": "B1"
    }
  ]
}

This is the expected (correct) output:

{
  "widgets": [
    {
      "barId": "B1"
    }
  ]
}

Why sometimes there is null elements in the widgets array and sometimes there is not? How I can avoid them?

Based on the many other SO questions the following operation:

{
  "operation": "modify-overwrite-beta",
  "spec": {
    "*": "=recursivelySquashNulls"
  }
}

should remove the null values, but why those are not removed in the following transformation?

[
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "type": {
            "BAR": {
              "@(2,id)": "widgets[&3].barId"
            }
          }
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": "=recursivelySquashNulls"
    }
  }
]

I'm observing this behavior in http://jolt-demo.appspot.com/

Answer 1

It looks like the web application in http://jolt-demo.appspot.com/ is running old version 0.1.1 that in this case behaves a bit differently than the newer versions.

At the moment the latest version available in Maven Central seems to be 0.1.7:

• https://central.sonatype.dev/artifact/com.bazaarvoice.jolt/jolt-core/0.1.7/versions

thought the Github release page only claims latest to be 0.1.6:

• https://github.com/bazaarvoice/jolt/releases/tag/jolt-0.1.6

I have the impression that the very powerful and useful Jolt library is not getting all the care it deserves :(

squashNull seems to work ok in Java code when using version 0.1.7.

Here is my final transformation that works as expected with http://jolt-demo.appspot.com/ based on answer by Barbaros Özhan:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "widgets": {
        "*": {
          "type": "=toLower"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "type": {
            "foo|bar": {
              "@(2,id)": "&4.&1.&1Id"
            }
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "@": "&2[]"
        }
      }
    }
  }
]

Answer 2

The first output keeps null as well(I don't know if it should appear in the desired output). Since the index(which is 1) of the object having "type": "ZAP" stays in the middle of the indexes(0,1,2), but not at the end, then doesn't vanish at that level.

Btw, squashNulls especially no impact for the objects nested within array. Indeed, it has some bugs and documented as fixed for version 0.1.6.

You can rather use the following spec which doesn't have square-bracketed indexes such as

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "type": {
            "FOO|BAR": {
              "@(2,id)": "&4.&1.&1Id"
            }
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "@": "&2[]"
        }
      }
    }
  }
]

Answer 3

This spec will help you resolve this query :

1)

Actually recursivelySquashNulls will work if you keep the array in a object for eg: I kept it under test object and made it to previous one.

This spec will resolve your issue :

 [
  {
    "operation": "shift",
    "spec": {
      "widgets": {
        "*": {
          "type": {
            "BAR": {
              "@(2,id)": "widgets[&3].barId"
            }
          }
        }
      }
    }
  }, {
    "operation": "shift",
    "spec": {
      "*": "test.&"
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": "=recursivelySquashNulls"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "test": {
        "*": "&"
      }
    }
  }
]