CODEWITHSUNDEEP
azureazure-data-factory
Godwin
Godwin

Reputation: 54

Creating JSON Array in Azure Data Factory with multiple Copy Activities output objects

Is it possible to embed the output of a copy activity in Azure Data Factory within an array that is meant to be iterated over in a subsequent ForEach?

My goal is to create an array with the output of several copy activities and then in a ForEach, access the properties of those copy activities with dot notation (Ex: item().rowsRead). Image shows code details.

Image

Specifically, I have 7 copy activities whose output JSON object (described here) would be stored in an array that I then iterate over. In the ForEach I would be checking the properties on each of the copy activities (rowsRead, rowsCopied, etc.) for validation purposes. https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-monitoring

Upvotes: 1

Views: 7760

Answers (1)

Joseph Xu
Joseph Xu

Reputation: 6083

I think we can embed the output of a copy activity in Azure Data Factory within an array. I've created a test to save the output of 2 Copy activities into an array. We need to concat a string type and then convert it to json type. Please see my step2.

enter image description here

  1. We can declare an array type variable named CopyInfo to store the output. The another array type variable named JsonArray is used to see the test result at debug mode.
    enter image description here

  2. In Append variable1 activity, I use @json(concat('{"activityName":"Copy1","activityObject":',activity('Copy data1').output,'}')) to save the output of Copy data1 activity and convert it from String type to Json type.
    enter image description here

  3. In Append variable2 activity, I use @json(concat('{"activityName":"Copy2","activityObject":',activity('Copy data2').output,'}')) to save the output of Copy data2 activity and convert it from String type to Json type.
    enter image description here

  4. Then I assign the value of variable CopyInfo to variable JsonArray enter image description here

  5. In the end, we can see the json array like :

"name": "JsonArray",
"value": [
    {
        "activityName": "Copy1",
        "activityObject": {
            "dataRead": 643,
            "dataWritten": 643,
            "filesRead": 1,
            "filesWritten": 1,
            ...
    },
    {
        "activityName": "Copy2",
        "activityObject": {
            "dataRead": 643,
            "dataWritten": 643,
            "filesRead": 1,
            "filesWritten": 1,
            ...
            }
    }
]

Upvotes: 2

Related Questions