CODEWITHSUNDEEP
azure-data-factory
Steve Powell
Steve Powell

Reputation: 21

Increasing concurrency in Azure Data Factory

We have a parent pipeline that gets a list of tables and feeds it into a ForEach. Within the ForEach we then call another pipeline passing in some config, this child pipeline moves the data for the table it is passed as config.

When we run this at scale I often see 20 or so instances of the child pipeline created in the monitor. All but 4 will be "Queued", the other 4 are executing as "In progress" . I can't seem to find any setting for this limit of 4. We have several hundred pipelines to execute and I really could do with it doing more than 4 at a time. I have set concurrency as 20 throughout the pipelines and tasks, hence we get 20 instances fired up. But I can't figure out what it is I need to twiddle to get more than 4 executing at the same time.

The ForEach looks like this

activities in ForEach loop look like this

many thanks

Upvotes: 0

Views: 6286

Answers (3)

Jatin
Jatin

Reputation: 111

In the ForEach activity by default 20 iterations can be executed in parallel. You can modify as per you requirements.

ForEach Activity

Another option you have is to set the concurrent executions of the pipelines by setting the value of concurrency property in the pipeline's setting tab. If you have triggered more than the specified concurrency setting, rest of the executions will be queued.

concurrency setting in pipeline

Upvotes: 0

Steve Powell
Steve Powell

Reputation: 21

I think I have found it. On the child Pipeline (the one that is being executed inside the ForEach loop) on the General Tab is a concurrency setting. I had this set to 4. When I increased this to 8 I got 8 executing, and when I increased it to 20 I got 20 executing.

Upvotes: 1

Frode
Frode

Reputation: 356

It seems max 20 loop iteration can be executed at once in parallel. The documentation is however a bit unclear.

The BatchCount setting that controls this have max value to 50, default 20. But in the documentation for isSequential it states maximum is 20.

Under Limitations and workarounds, the documentation states: "The ForEach activity has a maximum batchCount of 50 for parallel processing, and a maximum of 100,000 items."

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity

Upvotes: 0

Related Questions