CODEWITHSUNDEEP
azureazure-functionsazure-data-factory
Aleix
Aleix

Reputation: 461

How to re-try an ADF pipeline execution until conditions are met

An ADF pipeline needs to be executed on a daily basis, lets say at 03:00 h AM.

But prior execution we also need to check if the data sources are available.

Data is provided by an external agent, it periodically loads the corresponding data into each source table and let us know when this process is completed using a flag-table: if data source 1 is ready it set flag to 1.

I don't find a way to implement this logic with ADF.

We would need something that, for instance, at 03.00 h would trigger an 'element' that checks the flags, if the flags are not up don't launch the pipeline. Past, lets say, 10 minutes, check again the flags, and be like this for at most X times OR until the flags are up.

If the flags are up, launch the pipeline execution and stop trying to launch the pipeline any further.

How would you do it?

The logic per se is not complicated in any way, but I wouldn't know where to implement it. Should I develop an Azure Funtions that launches the Pipeline or is there a way to achieve it with an out-of-the-box AZDF activity?

Graphic description

Upvotes: 0

Views: 1844

Answers (1)

DraganB
DraganB

Reputation: 1138

There is a UNTIL iteration activity where you can check if your clause. Example:

  • Your azure function (AF) checking the flag and returns 0 or 1.
  • Build ADF pipeline with UNTIL activity where you check the output of AF (if its 1 do something). In UNTIL activity you can have your process step. For example, you have a variable flag that will before until activity is 0. In your until you check if it's 1. if it is do your processing step, if its not, put WAIT activity on 10 min or so.

So you have the ability in ADF to iterate until something it's not satisfied. Hope that this will help you :)

Upvotes: 2

Related Questions