CODEWITHSUNDEEP
azure-data-factory
NibblyPig
NibblyPig

Reputation: 52942

Is it possible to have more than 40 copy transform operations in Azure Data Factory ADF in a single pipeline?

I'm wondering if I'm doing this correctly as it seems odd.

I am doing a migration from one database to another. It has 90 tables.

I have generated the configuration for this programatically, so I edit the json files and commit/push them to the branch.

Initially, in the ADF pipeline, 90 copy operations appear in the pipeline along with an error saying the maximum allowed is 40.

I split it up into multiple pipelines. When I run the pipelines, one by one, they work correctly.

However it's a bit inconvenient to have three pipelines.

Is there a way to put more than 40 operations into a single pipeline? Or can I chain pipelines, or even have a pipeline of pipelines?

Upvotes: 0

Views: 317

Answers (1)

Abhishek Khandave
Abhishek Khandave

Reputation: 3230

Maximum activities per pipeline, which includes inner activities for containers is 40

enter image description here

But you can use Data Flow which doesn’t have limits on source and sink. You can create 40 Data Flow activities and can have more than 40 flows i.e. source to sink.

You can refer this similar link by Leon Yue

Reference - https://github.com/MicrosoftDocs/azure-docs/blob/main/includes/azure-data-factory-limits.md

Upvotes: 1

Related Questions