DIU does not increase not more than 4 on copy activity

Question

I am trying to copy data from GCP(Big Query) Azure Storage Gen2 parquet file with below configuration. Increased DIU from 4 to 16 but during runtime the DIU does not go beyond 4. Can you please help on how to increase the DIU to make my process faster?

1. using preserve hierarchy
2. Data size 12 millions with 3gb
3. Throughput is 2.5mbps

Answer 1

Sorry, I dont have enough reputations to write a comment so posting it as an answer if this helps.

Please read this thread. This is similar to yours with data size of 3GB. So I assume when your data size increase your DIU will also change.

Alternatively, you can increase the Degree of copy Parallelism (DoCP). I tried in my case for a 1.5GB dataset with source as ADLS and sink as Azure table. With default DoCP i.e. 4 the copy activity takes 10 minutes with throughput starting from 8Mbps and ends at 1.3Mbps. Where with DoCP as 16, the throughput ends with 2Mbps and it takes ~4 minutes to complete the copy activity. Both runs with a DIU of 4 units.

Answer 2

I have the same issue. My source is Azure Blob (csv files), Staging is also Azure Blob and final destination is Snowflake with Azure. All in the same region/zone.

I have set DIU for 20 and Parallel to 4. But only has DIU as 4 utilized and Parallel to 1.

Answer 3

To increase DIU for a copy activity just click on the activity, and under the Settings tab you can find the Data Integration Unit selector.