CODEWITHSUNDEEP
azureazure-data-factory
Neil P
Neil P

Reputation: 3190

Data Factory - is it possible to set a concurrency limit on a linked service?

I'm using Azure data factory with Azure Data Warehouse and I'd like to increase the Resource Class for my connection. In order to avoid tasks being forced to wait for concurrency slots to become available the database, I'd like to know if it is possible to tell data factory that it can only run 2 activities against a defined data source at a time. Hopefully, this should help to prevent any time outs on tasks that are suspended in the db waiting for concurrency slots to become available.

My data factory has two parallel pipelines that could potentially be using the same db at the same time.

Is it possible to set a global concurrency limit for a linked service?

Upvotes: 2

Views: 3855

Answers (1)

Paul Andrew
Paul Andrew

Reputation: 3253

In short, no.

Currently ADF concurrency is set only at an activity level.

Option 1

You could create some fake datasets to enforce a dependency within ADF to chain the activities across pipelines. Not ideal and a little messy.

Option 2

Limit the resource class for the ADF connecting user in SQL DW and let it handle the query queuing rather than ADF. Be careful of your ADF time out values though, if you do this ADF will kill executions if the time out is exceeded because it isn't aware of operations within the service its invoked.

Hope this helps.

Upvotes: 2

Related Questions