CODEWITHSUNDEEP
sqljsonazuresnowflake-cloud-data-platform
Kamran
Kamran

Reputation: 147

Parsing JSON in Snowflake

I'm trying to parse a the below nested JSON in Snowflake using the latteral function in Snowflake but I wanted to each nested column in "GoalTime" to show up as a column. For example, 

GoalTime_InDoorOpen         
2020-03-26T12:58:00-04:00   

GoalTime_InLastOff
null

GoalTime_OutStartBoarding
2020-03-27T14:00:00-04:00  

"GoalTime": [
    {
      "GoalName": "GoalTime_InDoorOpen",
      "GoalTime": "2020-03-26T12:58:00-04:00"
    },
    {
      "GoalName": "GoalTime_InLastOff"
    },
    {
      "GoalName": "GoalTime_InReadyToTow"
    },
    {
      "GoalName": "GoalTime_OutTowAtGate"
    },
    {
      "GoalName": "GoalTime_OutStartBoarding",
      "GoalTime": "2020-03-27T14:00:00-04:00"
    },

Upvotes: 1

Views: 3211

Answers (4)

sogurb
sogurb

Reputation: 21

The library below provides a method called "ExecuteAll" which one of the params is "tags", so if you provide an array of tags and values, all of them will be parsed and validated plus keeping the features of the sql injection protection from Snowflake.

snowflake-multisql

Upvotes: 0

Christian Screen
Christian Screen

Reputation: 56

Probably close to what you seek is using a standard SQL UNION statement.

Given the following are true to recreate the solution:

  • Created a table 'JSON_GOALS' with one column for raw JSON called, GOALS_RAW
  • You have loaded JSON data into a table as the raw JSON, with compliant JSON object array syntax, and a parent, GoalTimeGroup, ex: {[{}]}, so
{
    "GoalTimeGroup": [{
            "GoalName": "GoalTime_InDoorOpen",
            "GoalTime": "2020-03-26T12:58:00-04:00"
        },
        {
            "GoalName": "GoalTime_InLastOff"
        },
        {
            "GoalName": "GoalTime_InReadyToTow"
        },
        {
            "GoalName": "GoalTime_OutTowAtGate"
        },
        {
            "GoalName": "GoalTime_OutStartBoarding",
            "GoalTime": "2020-03-27T14:00:00-04:00"
        }
    ]
}

Doing so allows you to write a fairly standard JSON retrieve in Snowflake with the following syntax:

SELECT GOALS_RAW:GoalTimeGroup[0].GoalName, GOALS_RAW:GoalTimeGroup[1].GoalName, GOALS_RAW:GoalTimeGroup[2].GoalName
FROM JSON_GOALS 
UNION
SELECT GOALS_RAW:GoalTimeGroup[0].GoalTime, GOALS_RAW:GoalTimeGroup[1].GoalTime, GOALS_RAW:GoalTimeGroup[2].GoalName
FROM JSON_GOALS 
;

This gives you closer to the answer you are looking for and seems to provide a simpler solution. You can also control how many rows you'd want based on your JSON object attributes for each GOAL object.

Recommendations to enhance this would be to create a function that could detect the depth of each nested element and perhaps auto generate the indexes for 'n' number of columns.

Upvotes: 0

Simeon Pilgrim
Simeon Pilgrim

Reputation: 26078

or if you have many rows (what appear to be flights) and thus you need to columns per flight this code be what you are after

with data as (
select flight_code, parse_json(json) as json from values ('nz101','{GoalTime:[{"GoalName": "GoalA", "GoalTime": "2020-03-26T12:58:00-04:00"}, {"GoalName": "GoalB"}]}'),
                                                 ('nz201','{GoalTime:[{"GoalName": "GoalA"}, {"GoalName": "GoalB", "GoalTime": "2020-03-26T12:58:00-02:00"}]}') 
                                                 j(flight_code, json)
), unrolled as (
select d.flight_code, f.value:GoalName as goal_name, f.value:GoalTime as goal_time
from data d,
lateral flatten (input => json:GoalTime) f
)
select * 
from unrolled
    pivot(min(goal_time) for goal_name in ('GoalA', 'GoalB'))
order by flight_code;

it gives the results:

FLIGHT_CODE 'GoalA'                       'GoalB'
nz101       "2020-03-26T12:58:00-04:00"   null
nz201       null                          "2020-03-26T12:58:00-02:00"

Upvotes: 3

Greg Pavlik
Greg Pavlik

Reputation: 11086

create or replace function JSON_STRING()
returns string
language javascript
as
$$
return `
[
  {
        "GoalName": "GoalTime_InDoorOpen",
        "GoalTime": "2020-03-26T12:58:00-04:00"
    },
    {
        "GoalName": "GoalTime_InLastOff"
    },
    {
        "GoalName": "GoalTime_InReadyToTow"
    },
    {
        "GoalName": "GoalTime_OutTowAtGate"
    },
    {
        "GoalName": "GoalTime_OutStartBoarding",
        "GoalTime": "2020-03-27T14:00:00-04:00"
    }
]
`;
$$;

select value:GoalName::string as GoalName, value:GoalTime::timestamp as GoalTime
from lateral flatten(input => parse_json(JSON_STRING()));

-- See how the lateral flatten combination works on a JSON variant:
select * from lateral flatten(input => parse_json(JSON_STRING()));

I wrote this to run in any Snowflake worksheet, no tables needed. The function on top simply allows the JSON to be written as a multi-line string in the SQL statement below it. It has no other use than representing a string holding your JSON.

Step 1 is to PARSE_JSON, which converts a string into a variant data type formatted as a JSON object.

Step 2 is the lateral flatten. If you do a select star on that, it will return a number of columns. One of them is "value".

Step 3 is to extract the properties you want using single : notation for the property name and dots to traverse down the nodes from there (if there are any).

Step 4 is to cast the property to the data type you want using double :: notation. This is especially important if you're doing comparisons on the column particularly in join keys.

Note that there's a slight invalid part of the JSON that did not allow it to parse. In the top level the array had a property, which did not parse. I removed that to allow parsing.

Upvotes: 0

Related Questions