CODEWITHSUNDEEP
azureazure-data-factory
Akshay
Akshay

Reputation: 559

ADF Limitation : Number of COPY Data Activities in One Pipeline?

I have about 25 CSV Files with different columns (BLOB storage) with around ~250 columns each and want to load it Azure SQL DB separate tables(Basic Tier).

Created a Pipeline with 10 COPY Data Activity (CDA) all parallel in One pipeline for a start and executed it. The ADF pipeline just keeps on running without performing any task. When I reduce the CDA to 7, the pipeline works and loads the data in a mater of seconds. To check if there is any connections limitation with SQL database, executed 3 pipelines simultaneously with 7 CDA each and it worked.

Question here is --> Is there any Restriction/Limitation to the number of CDA we can have in a pipeline. If Yes, what can be done to change it ?

-Thanks

--EDIT Added Screenshot post applying solution provided to change property for parallel copies.

enter image description here

enter image description here

Upvotes: 1

Views: 2268

Answers (1)

CodeMonkey
CodeMonkey

Reputation: 477

Yes there are limitations. Since you are going from a Blob file store to an Azure SQL DB, to increase the amount of parallel copies you will need to set the parallelCopies property.

"activities":[
    {
        "name": "Sample copy activity",
        "type": "Copy",
        "inputs": [...],
        "outputs": [...],
        "typeProperties": {
            "source": {
                "type": "BlobSource",
            },
            "sink": {
                "type": "AzureSQLDBSink"
            },
            "parallelCopies": 32
        }
    }
]

From file store to non-file store - When copying data into Azure SQL Database or Azure Cosmos DB, default parallel copy also depend on the sink tier (number of DTUs/RUs).

  • When copying data into Azure Table, default parallel copy is 4.

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features

Upvotes: 2

Related Questions