CODEWITHSUNDEEP
azureazure-data-factory
Ryan Papke
Ryan Papke

Reputation: 165

Azure Data Factory V2 Trigger Pipeline only when current execution finished

Is there a way I can create a trigger in Azure Data Factory that will only kick my pipeline if it's not already running? I basically want to run a pipeline continuously and so for whatever reason it finished (failure / success) I want it to resume as soon as possible.

Upvotes: 6

Views: 5207

Answers (1)

Kranthi Pakala
Kranthi Pakala

Reputation: 1424

You could try creating a scheduled trigger with a recurrence of 1 minute (this is the minimum possible interval for a schedule trigger) and you can set pipeline Concurrency to 1. What this does is it will start the first run, then the trigger runs again and tries to run the pipeline, but since the first run hasn't finished, it will add it to the queue. After the first run finishes, it will start to run the second one.

enter image description here

enter image description here

Please note that there is a limit on Queue size of 100, so when using pipeline concurrency of 1 could have 1 pipeline run 'InProgress' and 100 in the 'Queue', after that, if they keep triggering they will get error 429.

Upvotes: 6

Related Questions