Azure Data Factory Tasks Queued

Question

I've created a new ADF pipeline which is working well but gives me some concern over performance.

As an example - here's a task from the pipeline that copies a small blob from a container to another container in the same storage account:

Notice that it's queued for 58 seconds.

The pipeline uses "Managed Virtual Network" integration runtime because it makes use of Azure SQL Private Endpoints.

Any ideas why the copy data tasks are held at "Queued" for so long?

Answer 1

As you mentioned that your pipeline using "Managed Virtual Network" integration runtime, therefore, as per the Activity execution time using managed virtual network:

By design, Azure integration runtime in managed virtual network takes longer queue time than global Azure integration runtime as we are not reserving one compute node per data factory, so there is a warm up for each activity to start, and it occurs primarily on virtual network join rather than Azure integration runtime. For non-copy activities including pipeline activity and external activity, there is a 60 minutes Time To Live (TTL) when you trigger them at the first time. Within TTL, the queue time is shorter because the node is already warmed up.

There is also 60 minutes time to Live(TTL) feature is available in "Managed Virtual Network" IR which shorten the queue time because the node is already warmed up, but unfortunately Copy activity doesn't have TTL support yet.