CODEWITHSUNDEEP
azureazure-data-factory
Hammad Hassan Khan
Hammad Hassan Khan

Reputation: 85

Rerunning a failed ADF pipeline automatically

I have multiple pipelines in Azure Data factory that get data from APIs and then push it to a datalake. I get alerts in case one of the pipelines fail. I then go to the ADF instance and rerun the the failed pipeline manually. I am trying to come up with an automated way of rerunning a pipeline in case it fails. Any suggestions or guidance would be helpful. thought of Azure logic apps or powerautomate but turns out don't have the right actions in there to trigger a failed pipeline.

Upvotes: 2

Views: 6806

Answers (3)

Hammad Hassan Khan
Hammad Hassan Khan

Reputation: 85

I came with a streamlined way of running failed pipelines. I decided to use the Azure Data factory API alongside Azure Logic apps to solve the problem.

I run logic apps on a scheduled run time and then use the following API commands:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/?api-version=2018-06-01

This API query gives us all the pipeine runs. If we want to filter it down to failed values, we can add the following body to it:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "2018-06-16T00:49:48.3686473Z",
  "filters": [
    {
      "operand": "status",
      "operator": "Equals",
      "values": [
        "failed"
      ]
    }
  ]
}

After getting the failed pipelines, We can then invoke the following API on each failed pipeline, to rerun them:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelines/{pipelineName}/createRun?api-version=2018-06-01

This solution can be created using a scripting language , powerautomate workflow or Azure Logic apps.

Upvotes: 2

Findo
Findo

Reputation: 21

If the pipeline design could be modified then a method can be to

  1. Set parameter pMax_rerun_count ( This is to ensure pipeline doesn go into indefinite loop )
  2. set 2 variables: (2.a) Pipeline_status default value : Fail (2.b) Max_loop_count default value : 0 ; This would be to ensure the pipeline doesnt run in loops . The value could be set during the pipeline run to have the maximum permissible retry count (i.e. pMax_rerun_count) passed as parameter in the pipeline
  3. All activities should be inside and Untill activity which will have expression or(equals(Pipeline_status,'Success'),equals(pMax_rerun_count,Max_loop_count)
  4. The first activity inside Untill activity will be Set Variable activity that increment the value of variable Max_loop_count by 1 .
  5. The final activity insisde Untill activity will be to Set variable activity that sets Pipeline_status to "Success"

The purpose here is to run all intended activities inside untill block untill the intended activities in pipeline completes successfully . pMax_rerun_count is to ensure pipeline doesnt go into indefinite loops.

This setup can considered as a framework if all pipelines needs to rerun in case of failure

Upvotes: 1

IpsitaDash-MT
IpsitaDash-MT

Reputation: 1450

As of now there is no inbuilt method to automate the process of "rerunning from failed activity" in the ADF, but each activity has a Retry option that you should certainly employ. In the pipeline, you may attempt any action as many times as necessary if it fails.

Allow the trigger to point to a new pipeline with a Execute activity that points to the current Azure Datafactory with the copy activity:

Then choose the Advanced -> Wait for completion option. After the execute pipeline is complete, the webhook action should contain logic to halt the DW.

Upvotes: 0

Related Questions