Azure data factory activity execute after all other copy data activities have completed

Question

I've got an Azure Data Factory V2 pipeline with multiple Copy Data activities that run in parallel.

I have a Pause DW web hook pauses an Azure data warehouse after each run. This activity is set to run after the completion of one of the longest running activities in the pipeline. The pipeline is set to trigger nightly.

Unfortunately, the time taken to run copy data activities varies because it depends on transactions that have been processed in the business, which varies each day. This means, I can't predict which activity of those that run in parallel will finish last. This means, often the whole pipeline fails because the DW has been paused before some of the activities have started.

What's the best way of running an activity only after all other activities in the pipeline have completed?

I have tried to add an If activity to the pipeline like this:

However, I then run into this error during validation:

If Condition1 The output of activity 'Copy small tables' can't be referenced since it has no output.

Does anyone have any idea how I can move this forwards?

thanks

Answer 1

I think you can use the Execute pipeline activity .

Let the trigger point to the new pipeline which has the "Execute activity " which points to the current ADF with the copy activity , please do select the option Advanced -> Wait for completion . Once the execute pipeline is done it should to move to the webhook activity which should have the logic to pause the DW .

Let me know how this goes .

Answer 2

Just orchestrate all your parallel activities towards PAUSE DWH activity. Then it will be executed after all your activities are completed.