CODEWITHSUNDEEP
jsonapache-nifijolt
GCL
GCL

Reputation: 45

Removing elements from JSON payload with JOLT transformation

I'm rather new to JOLT transformations so I'm still having trouble with pretty basic stuff.

I have a big JSON payload that I need to simplify by removing most data from it. The only attributes I want to keep are a normal attribute at the root level:

"root_element" : 14,

and a specific field from array elements:

"array_elements": [
  {
    "array_field": 103854
  },
  {
    "array_field": 9975
  },
  {
    "array_field": 47551
  }
],

It's impractical to use the "remove" operation since there are a large number of attributes at both root level and within array_elements.

If I use this JOLT transformation:

{
    "operation": "shift",
    "spec": {
      "array_elements": {
        "*": {
          "array_field": "[&1].array_field"
        }
      }
    }
 }

I get the following result:

[
  {
    "array_field": 103854
  },
  {
    "array_field": 9975
  },
  {
    "array_field": 47551
  }
]

Then I try to add the required root attribute to my transformation:

{
    "operation": "shift",
    "spec": {
    "root_element" : "root_element",
      "array_elements": {
        "*": {
          "array_field": "[&1].array_field"
        }
      }
    }
 }

But this only returns this:

{
  "root_element" : 14
}

How can I modify my JOLT transformation in order to generate this JSON?

{
 "root_element" : 14,
"array_elements":[
  {
    "array_field": 103854
  },
  {
    "array_field": 9975
  },
  {
    "array_field": 47551
  }
 ] 
}

there are more attributes in-between for my real case. As an Example:

{
  "root_element": 14,
  "another_root_element": "text",
  "array_elements": [
    {
      "array_field": 103854,
      "another_array_field": "more text"
    },
    {
      "array_field": 9975
    },
    {
      "array_field": 47551
    }
  ]
}

I need to remove the "another" elements but imagine there are multiple "another" elements (hundreds) with various names (too many and random). I just need to keep the attributes I described.

Thanks in advance!

Upvotes: 1

Views: 630

Answers (2)

Barbaros Özhan
Barbaros Özhan

Reputation: 65323

You can use the following transformation considering those other attributes :

[
  {
    "operation": "shift",
    "spec": {
      "root_element": "&", // pick attribute named "root_element" only
      "array_elements": {
        "*": {
          // pick attributes named "array_field" only
          "array_field": "&2[&1].&" // &2 grabs value after going two levels up the tree
                                    // &  replicates the current key-value combination
        }
      }
    }
  }
]

the combination "&2[&1].&" might also be replaced with "&2[#2].&" as anternative to suppress the null generation for same cases you may encounter.

Upvotes: 1

Mohammadreza Khedri
Mohammadreza Khedri

Reputation: 2691

You are trying to add array_field items to an array. I said that because you have written [&1].

So for root_element you want to add it to an object with the root_element key.

Now suppose, How JOLT can see that?

You write root_element as the first key you want to get in your JOLT spec, So JOLT creates an object for you and put root_element into that.

But for the next key (array_elements) you want to put that into an array but you have an object currently. So jolt ignore that key and create an object with just the root_element key in desired output for you.

For fixing that you can for example put array_elements in an object named arr, So you can have the following JOLT spec:

[
  {
    "operation": "shift",
    "spec": {
      "root_element": "root_element",
      "array_elements": {
        "*": {
          "array_field": "arr[&1].array_field"
        }
      }
    }
 }
]

Output:

{
  "root_element": 14,
  "arr": [
    {
      "array_field": 103854
    },
    {
      "array_field": 9975
    },
    {
      "array_field": 47551
    }
  ]
}

Upvotes: 1

Related Questions