CODEWITHSUNDEEP
azure-blob-storageazure-cosmosdbazure-data-factory
DxG
DxG

Reputation: 209

Increase parallel copy in Azure Data Factory

I'm trying to perform ADF copy activity from CosmosDB to Azure Blob Container. The degree of copy parallelism is always 1, even if I specify a greater value. what determines the parallelism value and how do I increase it ? . If the copy is from CosmosDB to another CosmosDB , I could achieve more than 1 parallel copies. But when the sink is blob container, I'm facing the issue.

Currently, the pipeline is writing into single file in Azure Blob Container. But I don't see an option to write multiple output files , incase if that helps to increase parallel copy.

Upvotes: 1

Views: 3872

Answers (1)

Pratik Lad
Pratik Lad

Reputation: 8291

As per the Official Microsoft Documentation,

The degree of copy parallelism in the copy activity means the maximum number of threads that copy activity uses to copy the data parallelly from the source to sink to increase the throughput.

It won’t generate multiple files in the sink.

It will only increase the throughput of the activity by copying concurrently using the number of threads we specify.

The copy activity only copies our source table or data to only single file despite the increase or decrease of degree of copy parallelism because these two actions have no relationship between them apart from the throughput.

If you want to copy multiple files from single cosmos db container, you can try a ForEach activity with certain inputs from cosmos db dataset with a copy activity inside it which copies the input data to a single file on each iteration of forEach. You can set the number of copies in this by giving value to Batch count of forEach.

ref1

Please go through this link from azurelib by Deepak Goyal to learn more about Parallel execution of copy activity using ForEach.

Upvotes: 2

Related Questions