Azure Data Factory: Get the result of a query on the databricks notebook to create a condition

Question

I wanted the result of a query on the databricks notebook to be the success or failure condition of the pipeline to reprocess for example the "copy data" in the azure data factory.

For example: If x = 1, terminate the pipeline, if not, reprocess (with a limit of 3 attempts).

What's the best way to do this?

Answer 1

You can do this with the help of if and until activities in ADF.

Please go through the sample demonstration below:

This is the sample Notebook code from databricks.

#your code
x=1
dbutils.notebook.exit(x)

In ADF, first create an array variable which will be used in the until activity.

This array length is used for n number of times re-process.

Next give your databricks notebook.

Now use an if activity and give the below expression in that.

@equals(activity('Notebook1').output.runOutput,1)

If this is true, our pipeline has to be terminated. So, add a fail activity in the True activities of if.

Here you can give any message that you want.

Leave the Fail activities of if as it is.

Now, use an until activity and give the success of if to it.

Inside Until activities we can give any activity. if you want to reprocess another pipeline then you can give execute pipeline also. Here I have given a copy activity. After copy activity use an append variable activity and give the array variable that we defined in the first and append with any single value that you want.

Now in the until expression give the below.

@equals(length(variables('iter')),4)

So, the activities inside until will reprocess 3 times if x!=1.

If x=1 in notebook, pipeline failed and terminated at if.

if x!=1 in Notebook, until reprocessed copy activity 3 times.