CODEWITHSUNDEEP
azureazure-data-factoryazure-databricks
MADFROST
MADFROST

Reputation: 1733

How to get output parameter from Executed Pipeline in ADF?

I have a databricks pipeline that will give an output, but at the moment, I need run the databricks from the Executed Pipelines, when I tried to run it, my databricks output didn't show up on Executed Pipelines ? Is this pipeline can't show the output ?

So this is my Databricks output result.

[![enter image description here][1]][1]

And this is my Executed Pipeline.

enter image description here

How can I get the runOutput result from Executed Pipeline ?

Upvotes: 7

Views: 10981

Answers (2)

Trent Tamura
Trent Tamura

Reputation: 1145

You can use the Azure REST API to query activity runs, so you could do an API call to get the last pipeline run of the pipeline you are interested in. Then you use those details to call another REST API to get the Activity output for that pipeline RunID you are interested in.

This method has the benefit of not needing to write output to a file or DB, rather you can just lookup the output of that activity in the Azure Monitor Logs for ADF. It can also be used for getting activity output from a completely different Data Factory.

You can actually refer to this post to show how to use the Azure REST API for ADF to query pipeline and activity runs:

Azure data factory and Log analytics

The important part here is the body and the operands:

Example Body in API Call

Here is the Microsoft Documentation on how to use the query pipeline API:

https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory

And here is the Microsoft documentation on how to query the activity API:

https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory

So what you can do is find the Pipeline RunID of the child pipeline that ran using the pipeline API, then use that to query for the specific activity execution within that pipeline using the Activity API.

Real Example:

Parent Pipeline:

Parent Pipeline

Child Pipeline:

Child Pipeline

To get the Last Pipeline Run of the Child Pipeline using Web Request:

Get Last Pipeline Run

URL:

https://management.azure.com/subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01

BODY:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [
    {
      "operand": "PipelineName",
      "operator": "Equals",
      "values": [
        "@{pipeline().parameters.PipelineName}"
      ]
    },
    {
      "operand": "LatestOnly",
      "operator": "Equals",
      "values": [
        true
      ]
    }
  ]
}

To get the Activity Output using the Pipeline details from Previous Web call:

Get Activity Output from Child

URL:

https://management.azure.com/subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelineruns/@{activity('Get Last ChildPipeline Run Details').output.value[0].runId}/queryActivityruns?api-version=2018-06-01

BODY:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [
    {
      "operand": "ActivityName",
      "operator": "Equals",
      "values": [
        "@{pipeline().parameters.ActivityName}"
      ]
    }
  ]
}

Here is the successful output in the Parent Pipeline using this process shown above:

Final output

Upvotes: 6

Abhishek Khandave
Abhishek Khandave

Reputation: 3228

Data bricks activity is inside child pipeline. So execute pipeline activity will not show output properties of child pipeline activites.

You can try this work around, You can write Child pipeline’s output to a database table. Output means A blob name or writing the parent runID to a SQL table.

By doing this parent pipeline will get reference to get the output.

Upvotes: 0

Related Questions