CODEWITHSUNDEEP
azureazure-blob-storage
bachr
bachr

Reputation: 6006

How to upload a 10 Gb file using SAS token

I'm trying to upload a large file (over 10Gb) to Azure Blob Storage using SAS tokens. I generate the tokens like this

    val storageConnectionString = s"DefaultEndpointsProtocol=https;AccountName=${accountName};AccountKey=${accountKey}"
    val storageAccount = CloudStorageAccount.parse(storageConnectionString)
    val client = storageAccount.createCloudBlobClient()

    val container = client.getContainerReference(CONTAINER_NAME)
    val blockBlob = container.getBlockBlobReference(path)
    val policy = new SharedAccessAccountPolicy()
    policy.setPermissionsFromString("racwdlup")
    val date = new Date().getTime();
    val expiryDate = new Date(date + 8640000).getTime()
    policy.setSharedAccessStartTime(new Date(date))
    policy.setSharedAccessExpiryTime(new Date(expiryDate))
    policy.setResourceTypeFromString("sco")
    policy.setServiceFromString("bfqt")
    val token = storageAccount.generateSharedAccessSignature(policy)

Then I tried the Put Blob API and hit the following error

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: BlockBlob' -F [email protected]  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=.... -v

< HTTP/1.1 413 The request body is too large and exceeds the maximum permissible limit.
< Content-Length: 290
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: f08a1473-301e-006a-4423-837a27000000
< x-ms-version: 2019-02-02
< x-ms-error-code: RequestBodyTooLarge
< Date: Sat, 05 Sep 2020 01:24:35 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:f08a1473-301e-006a-4423-837a27000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:24:35.7712576Z</Message><MaxLimit>268435456</MaxLimit></Error>%

After that tried uploading it using PageBlob (I saw in the documentation something like size can be up to 8 TiB)

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: PageBlob' -H 'x-ms-blob-content-length: 1099511627776' -F [email protected]  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=... -v

< HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
< Content-Length: 331
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: b00d5c32-101e-0052-3125-83dee7000000
< x-ms-version: 2019-02-02
< x-ms-error-code: InvalidHeaderValue
< Date: Sat, 05 Sep 2020 01:42:24 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:b00d5c32-101e-0052-3125-83dee7000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:42:24.5137237Z</Message><HeaderName>Content-Length</HeaderName><HeaderValue>10114368132</HeaderValue></Error>%

Not sure what is the proper way to go about uploading such large file?

Upvotes: 2

Views: 1110

Answers (2)

Shiraz Bhaiji
Shiraz Bhaiji

Reputation: 65431

To copy large files to a blob you can use azcopy:

Authenticate first:

azcopy login

Then copy the file:

azcopy copy 'C:\myDirectory\myTextFile.txt' 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt'

https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs?toc=/azure/storage/blobs/toc.json

Upvotes: 0

Athanasios Kataras
Athanasios Kataras

Reputation: 26432

Check the different blob types here: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

The page blob actually limits the maximum size to 8TB but it's optimal for for random read and write operation.

On the other hand:

Block blobs are optimized for uploading large amounts of data efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. A block blob can include up to 50,000 blocks.

So block blobs is the way to go as it supports sizes of up to 190.7 TB (preview mode)

Now you need to use the put block https://learn.microsoft.com/en-us/rest/api/storageservices/put-block to upload the blocks that will form your blob.

Upvotes: 1

Related Questions