Copying multiple files from Blob storage to Azure SQL Database

Question

I have a question for which I am unable to get clarity, hoping you all could share some insights on this.

I read at few places that while copying multiple files we can use for each activity which copies all files in parallel. I want to understand if it would really copy the files in parallel, because my understanding is that for each activity activity would pick 1 first file load it and then move to the second file for copying.

Please advise, if my understanding is wrong.

Thanks,

Answer 1

I'm assuming the Copy activity is an Activity inside the ForEach Activity. If so, the ForEach activity has a "Sequential" checkbox:

If the box is checked, it will only process one iteration at a time. If it is unchecked, you can specify the Batch count which will determine how many iterations to run simultaneously:

The DEFAULT value (if unspecified) is 20. The MAX allowed value is 50. So if you are trying to copy 100s of files, it will not be "all" at the same time.

NOTE: It needs to be clear that the iterations are whatever collection of Activities are inside the ForEach. This is especially important if the iteration updates Variables because they are global to the pipeline and unexpected results may occur when they are updated in parallel runs.

If you are trying to copy an entire folder, there is no need to reference individual files. You can copy entire containers or container/directory by specifying parameter values in the Dataset (Binary recommended):

Then reference those in the Connection tab:

Then supply the values in your activity configuration:

BONUS: If you are copying within the same Storage Account (Blob or ADLS), you can use the same Dataset for Source and Sink.