CODEWITHSUNDEEP
jsongoogle-bigquery
Tajpi
Tajpi

Reputation: 35

Importing a nested json as a table with multiple nesting

In our data we have json fields that include repeated sections, as well as infinite nesting possibilities (the samples I have so far are quite simplistic). After seeing BQ repeated fields and records, I decided to try restructuring the data into repeated record fields, as our use case is related to analytics and then wanted to test out different use cases for the data to see which approach is more efficient (time/cost/difficulty) for the analysis we intend to do on it. I have created a sample json record that I want to upload to BQ, that uses all the features that I think we would need (I have validated is using http://jsonlint.com/):

{
    "aid": "6dQcrgMVS0",
    "hour": "2016042723",
    "unixTimestamp": "1461814784",
    "browserId": "BdHOHp2aL9REz9dXVeKDaxdvefE3Bgn6NHZcDQKeuC67vuQ7PBIXXJda3SOu",
    "experienceId": "EXJYULQOXQ05",
    "experienceVersion": "1.0",
    "pageRule": "V1XJW61TPI99UWR",
    "userSegmentRule": "67S3YVMB7EMQ6LP",
    "branch": [{
        "branchId": "1",
        "branchType": "userSegments",
        "itemId": "userSegment67S3YVMB7EMQ6LP",
        "headerId": "null",
        "itemMethod": "null"
    }, {
        "branchId": "1",
        "branchType": "userSegments",
        "itemId": "userSegment67S3YVMB7EMQ6LP",
        "headerId": "null",
        "itemMethod": "null"
    }],
    "event": [{
        "eventId": "546",
        "eventName": "testEvent",
        "eventDetails": [{
            "key": "a",
            "value": "1"
        }, {
            "key": "b",
            "value": "2"
        }, {
            "key": "c",
            "value": "3"
        }]
    }, {
        "eventId": "547",
        "eventName": "testEvent2",
        "eventDetails": [{
            "key": "d",
            "value": "4"
        }, {
            "key": "e",
            "value": "5"
        }, {
            "key": "f",
            "value": "6"
        }]
    }]
}

I am using BQ interface, to upload this json into a table with the following structure:

[
    {
        "name": "aid",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "hour",
        "type": "INTEGER",
        "mode": "NULLABLE"
    },
    {
        "name": "unixTimestamp",
        "type": "INTEGER",
        "mode": "NULLABLE"
    },
    {
        "name": "browserId",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "experienceId",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "experienceVersion",
        "type": "FLOAT",
        "mode": "NULLABLE"
    },
    {
        "name": "pageRule",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "userSegmentRule",
        "type": "STRING",
        "mode": "NULLABLE"
    },
    {
        "name": "branch",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
            {
                "name": "branchId",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "branchType",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "itemId",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "headerId",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "itemMethod",
                "type": "STRING",
                "mode": "NULLABLE"
            }
        ]
    },
    {
        "name": "event",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
            {
                "name": "evenId",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "eventName",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "eventDetails",
                "type": "RECORD",
                "mode": "REPEATED",
                "fields": [
                    {
                        "name": "key",
                        "type": "STRING",
                        "mode": "NULLABLE"
                    },
                    {
                        "name": "value",
                        "type": "STRING",
                        "mode": "NULLABLE"
                    }
                ]
            }
        ]
    }
]

My jobs fail with a 

JSON parsing error in row starting at position 0 in <file_id>. Expected key (error code: invalid)

It is possible I can't have multiple nesting in a table, but the error seems more as if there was an issue with parsing the JSON itself. I was able to generate and successfully import a json with a simple repeated record (see example below):

{
"eventId": "546",
"eventName": "testEvent",
"eventDetails": [{
    "key": "a",
    "value": "1"
}, {
    "key": "b",
    "value": "2"
}, {
    "key": "c",
    "value": "3"
}]

}

Any advice is appreciated.

Upvotes: 0

Views: 2245

Answers (1)

Danny Kitt
Danny Kitt

Reputation: 3251

There doesn't seem to be anything problematic with your schema, so BigQuery should be able to load your data with your schema.

First, make sure you are uploading newline-delimited JSON to BigQuery. Your example row has many newline characters in the middle of your JSON row, and the parser is trying to interpret each line as a separate JSON row.

Second, it looks like your schema has the key "evenId" in the "event" record, but your example row has the key "eventId".

Upvotes: 2

Related Questions