CODEWITHSUNDEEP
azure-data-factory
santosh sharma
santosh sharma

Reputation: 11

JSon Parsing in ADF web activitiy

I am trying to use the JSON out from one activity to another post activity

Output fro the 1st activity is

{ "pid": "0", "name": " test_onbirded1", "description": " ON_BOARDED_TRAIT111 ", "traitType": " ON_BOARDED_TRAIT", "dataSourceId": " 122233", "folderId": " 982331", "traitRule": "(lifecyclemainfrequent == 2)", "ADFWebActivityResponseHeaders": { "x-ms-request-id": "b583f633-501e-0089-1270-7961c2000000", "x-ms-version": "2020-08-04", "x-ms-creation-time": "Tue, 13 Jul 2021 13:18:52 GMT", "x-ms-lease-status": "unlocked", "x-ms-lease-state": "available", "x-ms-blob-type": "BlockBlob", "x-ms-server-encrypted": "true", "Accept-Ranges": "bytes", "Date": "Thu, 15 Jul 2021 11:58:36 GMT", "ETag": ""0x8D94787B0F9ED99"", "Server": "Windows-Azure-Blob/1.0;Microsoft-HTTPAPI/2.0", "Content-Length": "208", "Content-Type": "application/octet-stream", "Last-Modified": "Thu, 15 Jul 2021 11:57:16 GMT" }, "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (North Europe)", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 0 }, "billingReference": { "activityType": "ExternalActivity", "billableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }

But I only want to use

{ "pid": "0", "name": " test_onbirded1", "description": " ON_BOARDED_TRAIT111 ", "traitType": " ON_BOARDED_TRAIT", "dataSourceId": " 122233", "folderId": " 982331", "traitRule": "(lifecyclemainfrequent == 2)}

in the next web activity which is post

How do I exclude the other values from the output to the input :) to next web activity.

Upvotes: 0

Views: 1668

Answers (3)

Abhishek Khandave
Abhishek Khandave

Reputation: 3230

You cannot achieve this in Data Flow.

So, in Copy Data activity, use the column mapping to generate a new file (in sink).

Used this new file as source in the Data Flow.

Step1: Mapping:

enter image description here

Step2: While Mapping delete unnecessary properties as shown in below image:

enter image description here

Step3: Input

enter image description here

Step4: Output

enter image description here

Upvotes: 0

Trent Tamura
Trent Tamura

Reputation: 1145

Rather than using a data flow and spinning up an entire spark cluster just to make an API Call, it's probably better to explicitly set the body of the next API using the output value from previous API call directly.

For example:

if your response from the first API call is:

{ "pid": "0", "name": " test_onbirded1", "description": " ON_BOARDED_TRAIT111 ", "traitType": " ON_BOARDED_TRAIT", "dataSourceId": " 122233", "folderId": " 982331", "traitRule": "(lifecyclemainfrequent == 2)", "ADFWebActivityResponseHeaders": { "x-ms-request-id": "b583f633-501e-0089-1270-7961c2000000", "x-ms-version": "2020-08-04", "x-ms-creation-time": "Tue, 13 Jul 2021 13:18:52 GMT", "x-ms-lease-status": "unlocked", "x-ms-lease-state": "available", "x-ms-blob-type": "BlockBlob", "x-ms-server-encrypted": "true", "Accept-Ranges": "bytes", "Date": "Thu, 15 Jul 2021 11:58:36 GMT", "ETag": ""0x8D94787B0F9ED99"", "Server": "Windows-Azure-Blob/1.0;Microsoft-HTTPAPI/2.0", "Content-Length": "208", "Content-Type": "application/octet-stream", "Last-Modified": "Thu, 15 Jul 2021 11:57:16 GMT" }, "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (North Europe)", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 0 }, "billingReference": { "activityType": "ExternalActivity", "billableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }

then the body of your next API call should be like so:

{ 
    "pid": "@{activity('First API Call').output.pid}",
    "name": "@{activity('First API Call').output.name}",
    "description": "@{activity('First API Call').output.description}",
    "traitType": "@{activity('First API Call').output.traitType}",
    "dataSourceId": "@{activity('First API Call').output.dataSourceId}", 
    "folderId": " @{activity('First API Call').output.folderId}", 
    "traitRule": "@{activity('First API Call').output.traitRule}"
}

As you can see it is using the output of the previous activity directly, without the cost of spinning up a spark cluster.

Upvotes: 1

Luv Raheja
Luv Raheja

Reputation: 1

Use Select as an interim activitym and then use post, if that is possible. https://learn.microsoft.com/en-us/azure/data-factory/data-flow-select

Upvotes: 0

Related Questions