CODEWITHSUNDEEP
azureazure-data-factory
Shivshankar Kanawade
Shivshankar Kanawade

Reputation: 9

Azure Data Factory Increase copy speed

I do have an ADF data pipeline where I am copying a single file ~ 2 TB from Gen1 to Gen2 storage I have set the DTU to 32 and degree of parallelism to high but still its using 4 DTU and 1 parallel copy. The file in Gen1 storage is in structured stream format and it copies in Gen2 in parquet format

Its taking a lot of time around 20 hours any thoughts how we can increase the speed of it. I do have some file that are in ~100 tb in size so what can be a good strategy to do it.enter image description here

Did try to set the DTU to 32 and degree of parallelism to high but didnt improve performance

Upvotes: 0

Views: 23

Answers (0)

Related Questions