CODEWITHSUNDEEP
arraysjsonazure-data-factory
Psihiatr
Psihiatr

Reputation: 128

Azure Data Factory - skipped rows with empty collection reference when flattening JSON

I am trying to serialize the following JSON to Parquet using ADF Copy Activity:

[
    {
        "mac": "06A1E8A75834",
        "timestamp": "2020-11-06T00:00:00+02:00",
        "floor_number": 2,
        "x": 300.00,
        "y": 350.00,
        "located_inside": true,
        "zones": [
            {
                "zone_map_name": "sections",
                "zone_name": "z_1_"
            }
        ]
    },
    {
        "mac": "06A1E8A75835",
        "timestamp": "2020-11-06T00:00:00+02:00",
        "floor_number": 2,
        "x": 300.00,
        "y": 300.00,
        "located_inside": true,
        "zones": []
    }
]

However in some cases the zones array is empty and the rest of the record is skipped. The expected behavior is when the array is empty - the zone_name and zone_map_name are empty.

The mapping looks like this: mapping

Is there a way to tell ADF how to treat empty collection reference as empty values in rows?

Upvotes: 2

Views: 1365

Answers (1)

Joseph Xu
Joseph Xu

Reputation: 6043

As we know, ADF will skipped rows with empty collection reference when flattening JSON.
When we uncheck that box. enter image description here

ADF will copy all the items to destination, like this:

[{"mac":"06A1E8A75834","timestamp":"2020-11-06T00:00:00+02:00","floor_number":2,"x":300.0,"y":350.0,"located_inside":true,"zone_map_name":"sections","zone_name":"z_1_"}
,{"mac":"06A1E8A75835","timestamp":"2020-11-06T00:00:00+02:00","floor_number":2,"x":300.0,"y":300.0,"located_inside":true}
]

Upvotes: 1

Related Questions