CODEWITHSUNDEEP
azure-databricksazure-data-factory
harshith
harshith

Reputation: 91

how to get the name of the activity dynamically in ADF

I have a pipeline which takes the name of all stages and use it in the notebook, how to fetch the name of each stage dynamically in ADF and use it? how to achieve this case?

Upvotes: 3

Views: 4955

Answers (1)

Rakesh Govindula
Rakesh Govindula

Reputation: 11234

Currently, it is not possible in ADF to get the activity name dynamically. But, as you want the failed activity name, we can get the name from the error itself (in this case).

Please follow the demonstration below:

First, treat your main pipeline as a child pipeline and execute it using a parent pipeline and use the parent pipeline to get the activity name and error in the child pipeline.

This is my sample child pipline (main pipeline). In this, to create an error I gave wrong source file path in the copy activity named as My Copy Data.

enter image description here

Use Set Variable to store the error and activity name by which the error was generated after the execute pipeline activity in the parent pipeline.

@activity('Execute Pipeline1').error.message this will give you error like Operation on target activity failed. By using this we can get name of the activity in next variable.

Set variable activity for Error:


enter image description here
Set Variable for activity name:
Use this expression to get activity name. 

@substring(variables('error'),  add(indexOf(variables('error'),'target'),7),  sub(indexOf(variables('error'),'failed'),add(indexOf(variables('error'),'target'),7)))



enter image description here
Databricks Notebook activity:

create a databricks notebook activity and pass these variables as parameters to it and store it in the delta table.

enter image description here
Code in databricks notebook: 

activity_name=dbutils.widgets.get("activity_name")
error=dbutils.widgets.get("error")

#Create a delta table before this and append these values on delta table for 
every failure

print(activity_name)
print(error)

Databricks Job after execution:

enter image description here


You can get mail by using a logic app after Notebook activity and pass the activity name and error message to it using web activity.

First create a logic app for mail by following this Official Microsoft documentation.

Then, give the URL of logic app in web activity and body like below. 

{

"message" : "The failed activity in your pipeline is @{variables('activityname')}

and This is the error message @{variables('error')}.",

"dataFactoryName" : "@{pipeline().DataFactory}",

"pipelineName" : "@{pipeline().Pipeline}",

"receiver" : "@{pipeline().parameters.reciever}"

}



enter image description here

Now you can get the mail for every failure of your child pipeline.

Upvotes: 2

Related Questions