Jolt: Merge arrays from properties

Question

I'm trying to extract and merge objects from an array contained in some (but not all) of my input elements. Using the JOLT JSON transformation library.

Also, the arrays I'm trying to merge contain objects that don't always have the same properties. One key might be present in some, but not others.

Example is contrived/nonsensical simplification, but has the general shape of our data.

Input:

{
  "Widgets": [
    {
      "Id": "1",
      "PetFriendly": "True",
      "Features": [
        {
          "Name": "Easy Button",
          "Type": "Button"
        },
        {
          "Name": "Lunch Lever",
          "Type": "Food Service",
          "MenuItems": [
            "Pizza",
            "Cheezburger"
          ]
        }
      ]
    },
    {
      "Id": "2",
      "PetFriendly": "True"
    },
    {
      "Id": "3",
      "PetFriendly": "False",
      "Features": [
        {
          "Name": "Missles",
          "Type": "Attack"
        }
      ]
    },
    {
      "Id": "4",
      "PetFriendly": "False",
      "Features": [
        {
          "Name": "Bombs",
          "Type": "Attack",
          "MenuItems": [
            "Rat Poison"
          ]
        }
      ]
    }
  ]
}

Desired output:

  {
    "Widgets": [
      {
        "Id": "1"
        "PetFriendly": "True"
      },
      {
        "Id": "2"
        "PetFriendly": "True"
      },
      {
        "Id": "3",
        "PetFriendly": "False"
      },
      {
        "Id": "4",
        "PetFriendly": "False"
      }
    ],
    "Features": [
      {
        "WidgetId": "1",
        "Name": "Easy Button",
        "Type": "Button"
      },
      {
        "WidgetId": "1",
        "Name": "Lunch Lever",
        "Type": "Food Service",
        "MenuItems": [
            "Pizza",
             "Cheezburger"
         ]
      },
      {
        "WidgetId": "3",
        "Name": "Missles",
        "Type": "Attack"
      },
      {
        "WidgetId": "4",
        "Name": "Bombs",
        "Type": "Attack",
        "MenuItems": [
          "Rat Poison"
        ]
      }
    ]
  }

I have tried many transforms with no success, and read all the ShiftR documentation and its unit tests. A little help?

Answer 1

Spec

[
  {
    "operation": "shift",
    "spec": {
      "Widgets": {
        "*": {
          // build the finished "Widgets" output
          "Id": "Widgets[&1].Id",
          "PetFriendly": "Widgets[&1].PetFriendly",
          // 
          // Process the Features, by pushing the Id
          //  down into them, but maintain the same doubly
          //  nested structure.
          // Shift works property by property, so first 
          //  fix the properties in side each Features element,
          //  (pulling ID down).
          // Then in a 2nd Shift can accumulate things into array. 
          "Features": {
            "*": {
              "@(2,Id)": "temp[&3].Features[&1].WidgetId",
              "*": "temp[&3].Features[&1].&"
            }
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      // passthru
      "Widgets": "Widgets",
      "temp": {
        "*": {
          "Features": {
            // walk thru the doubly nested structure an
            //  now accumulate all non-null itens into 
            //  the the final Features array.
            "*": "Features[]"
          }
        }
      }
    }
  }
]

Answer 2

Finally got it working with the below spec, BUT it has an undesirable side effect: It leaves empty default arrays. Is there a way to remove empty arrays, or otherwise mark them during the default step so they can be deleted? I checked this GitHub issue but not sure how to translate it to arrays of string. Anyone have a better solution?

[
  // First fill in default value for "MenuItems" since not all Features have it.
  {
    "operation": "default",
    "spec": {
      "Widgets[]": {
        "*": {
          "Features[]": {
            "*": {
              "MenuItems": []
            }
          }
        }
      }
    }
  },
  {
    // Extract the Features' properties into arrays. The defaults added above ensure that we can merge the arrays into Feature objects as in this example: 
    // https://github.com/bazaarvoice/jolt/blob/master/jolt-core/src/test/resources/json/shiftr/mergeParallelArrays2_and-do-not-transpose.json.
    "operation": "shift",
    "spec": {
      "Widgets": {
        "*": {
          "Id": "Widgets[&1].Id",
          "PetFriendly": "Widgets[&1].PetFriendly",
          "Features": {
            "*": {
              "@(2,Id)": "temp.WidgetId",
              "Name": "temp.Name",
              "Type": "temp.Type",
              "MenuItems": "temp.MenuItems[]"
            }
          }
        }
      }
    }
  },
  // Finally merge the arrays into Feature objects.
  {
    "operation": "shift",
    "spec": {
      "Widgets": "Widgets",
      "temp": {
        "WidgetId": {
          "*": "Features[&0].WidgetId"
        },
        "Name": {
          "*": "Features[&0].Name"
        },
        "Type": {
          "*": "Features[&0].Type"
        },
        "MenuItems": {
          "*": "Features[&0].MenuItems"
        }
      }
    }
  }
]

Result:

{
  "Widgets": [
    {
      "Id": "1",
      "PetFriendly": "True"
    },
    {
      "Id": "2",
      "PetFriendly": "True"
    },
    {
      "Id": "3",
      "PetFriendly": "False"
    },
    {
      "Id": "4",
      "PetFriendly": "False"
    }
  ],
  "Features": [
    {
      "WidgetId": "1",
      "Name": "Easy Button",
      "Type": "Button",
      "MenuItems": []
    },
    {
      "WidgetId": "1",
      "Name": "Lunch Lever",
      "Type": "Food Service",
      "MenuItems": [ "Pizza", "Cheezburger" ]
    },
    {
      "WidgetId": "3",
      "Name": "Missles",
      "Type": "Attack",
      "MenuItems": []
    },
    {
      "WidgetId": "4",
      "Name": "Bombs",
      "Type": "Attack",
      "MenuItems": [ "Rat Poison" ]
    }
  ]
}