CODEWITHSUNDEEP
azureazure-storageazure-blob-storageazure-sdk-.net
user13308826
user13308826

Reputation:

How to upload a large file in chunks with parallelism in Azure SDK v12?

In Azure SDK v11, we had the option to specify the ParallelOperationThreadCount through the BlobRequestOptions. In Azure SDK v12, I see that the BlobClientOptions does not have this, and the BlockBlobClient (previously CloudBlockBlob in Azure SDK v11), there is only mention of parallelism in the download methods.

We have three files: one 200MB, one 150MB, and one 20MB. For each file, we want the file to be split into blocks and have those uploaded in parallel. Is this automatically done by the BlockBlobClient? If possible, we would like to do these operations for the 3 files in parallel as well.

Upvotes: 4

Views: 7964

Answers (2)

Ivan Glasenberg
Ivan Glasenberg

Reputation: 29950

You also can take use of StorageTransferOptions in v12.

The sample code below:

   BlobServiceClient blobServiceClient = new BlobServiceClient(conn_str);
   BlobContainerClient containerClient= blobServiceClient.GetBlobContainerClient("xxx"); 
   BlobClient blobClient = containerClient.GetBlobClient("xxx");

   //set it here.
   StorageTransferOptions transferOptions = new StorageTransferOptions();
   //transferOptions.MaximumConcurrency or other settings.       

   blobClient.Upload("xxx", transferOptions:transferOptions);

By the way, for uploading large files, you can also use Microsoft Azure Storage Data Movement Library for better performance.

Upvotes: 2

user13308826
user13308826

Reputation:

Using Fiddler, I verified that BlockBlobClient does indeed upload the files in chunks without needing to do any extra work. For doing each of the major files in parallel, I simply had a task for each one, added it to a list tasks and used await Task.WhenAll(tasks).

Upvotes: 0

Related Questions