Azure Data Factory: can I pause a Pipeline run while another Pipeline is running and then resume the first Pipeline?

Question

So I have this Pipeline that runs for a long time (weeks), which loads some tables with Data Factory and processes them with Databricks.

Also, I have another Pipeline that is run each day for a couple of hours. However, the Databricks cluster seems not to be powerful enough to run both pipelines simultaneously, as it throws an error when both Pipelines are active (seems to be a memory error, "Spark driver has stopped unexpectedly").

The daily Pipeline is highest priority, though, so ideally I would like to pause for around 3 hours the long term Pipeline, then execute the daily trigger, and then resume the long term Pipeline execution.

Is it possible to do that?

Thanks in advance!

Answer 1

You cannot pause a running pipeline. Azure devops pipeline doesnot have this feature currently. Below screenshots lists all the statuses that a build could be. As you can see "pause" is not one of them.

You might have to ask for a bigger capacity to run another cluster.

However, you can always submit a feature request(Click Suggest a feature and choose Azure Devops) to Microsoft Development team. Hope they will consider implementing this feature in the future sprints.

Answer 2

The simple answer: No. You should look for the solution in the databricks cluster throwing the error. There is no way to pause a pipeline because another pipeline in running.