CODEWITHSUNDEEP
azureazure-data-factory
Heather Sawatsky
Heather Sawatsky

Reputation: 362

How to get custom output from an executed pipeline?

I would like to be able to get custom output from an "Execute Pipeline Activity". During the execution of the invoked pipeline, I capture some information in a variable using the "Set Variable" activity. I would like to be able to use that value in the master pipeline.

I know that the master pipeline can read the invoked pipeline's name and runId using "@activity('InvokedPipeline').output," but those are the only properties available.

I have the invokable pipeline because it's configurable to be used by multiple other pipelines, assuming we can get the output from it. It currently consists of 8 activities; I would hate to have to duplicate them all across multiple pipelines just because we can't get the output from an invoked pipeline.

Reference: Execute Pipeline Activity

[
  {
    "name": "MasterPipeline",
    "type": "Microsoft.DataFactory/factories/pipelines"
    "properties": {
        "description": "Uses the results of the invoked pipeline to do some further processing",
        "activities": [
            {
                "name": "ExecuteChildPipeline",
                "description": "Executes the child pipeline to get some value.",
                "type": "ExecutePipeline",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "pipeline": {
                        "referenceName": "InvokedPipeline",
                        "type": "PipelineReference"
                    },
                    "waitOnCompletion": true
                }
            },
            {
                "name": "UseVariableFromInvokedPipeline",
                "description": "Uses the variable returned from the invoked pipeline.",
                "type": "Copy",
                "dependsOn": [
                    {
                        "activity": "ExecuteChildPipeline",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ]
            }
        ],
        "parameters": {},
        "variables": {}
    }
  },
  {
    "name": "InvokedPipeline",
    "type": "Microsoft.DataFactory/factories/pipelines"
    "properties": {
        "description": "The child pipeline that makes some HTTP calls, gets some metadata, and sets a variable.",
        "activities": [
            {
                "name": "SetMyVariable",
                "description": "Sets a variable after some processing from other activities.",
                "type": "SetVariable",
                "dependsOn": [
                    {
                        "activity": "ProcessingActivity",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "MyVariable",
                    "value": {
                        "value": "@activity('ProcessingActivity').output",
                        "type": "Expression"
                    }
                }
            }
        ],
        "parameters": {},
        "variables": {
            "MyVariable": {
                "type": "String"
            }
        }
    }
  }
]

Upvotes: 14

Views: 10498

Answers (2)

brajesh jaishwal
brajesh jaishwal

Reputation: 347

just had a chat with ADF team, and the response

[10:11 PM] Mark Kromer Brajesh Jaishwal: any plans on custom output from execute pipeline activity? Yes, this work is on the engineering work plan

Upvotes: 0

MartinJaffer-MSFT
MartinJaffer-MSFT

Reputation: 728

Hello Heather and thank you for your inquiry. Custom outputs are not an inbuilt feature at this time. You can request/upvote for the feature in the Azure feedback forum. For now, I do have two workarounds.

Utilizing the invoked pipeline's runID, we can query the REST API (using Web Activity) for the activity run logs, and from there, the activity outputs. However, before making the query, it is necessary to authenticate. REST call to get the activities of a pipeline

For authentication I reccomend using the Web Activity to get an oauth2 token. The URL would be https://login.microsoftonline.com/tenantid/oauth2/token. Headers "Content-Type": "application/x-www-form-urlencoded" and body "grant_type=client_credentials&client_id=xxxx&client_secret=xxxx&resource=https://management.azure.com/". Since this request is to get credentials, the Authentication setting for this request is type 'None'. These credentials correspond to an App you create via Azure Active Directory>App Registrations. Do not forget to assign the app RBAC in Data FActory Access Control (IAM).

Another workaround, has the child pipeline write its output. It can write to a database table, or it can write to a blob (I passed the Data Factory variable to a Logic App which wrote to blob storage), or to something else of your choice. Since you are planning to use the child pipeline for many different parent pipelines, I would recommend passing the child pipeline a parameter that it uses to identify the output to the parent. That could mean a blob name, or writing the parent runID to a SQL table. This way the parent pipeline knows where to look to get the output.

Upvotes: 12

Related Questions