CODEWITHSUNDEEP
pythonflaskazure-functionsazure-web-app-serviceazure-blob-storage
yihao.fu
yihao.fu

Reputation: 143

Upload large blob to Azure storage container using App service and function app

I am working on a project to allow users to upload blob into blob container in our storage account. I developed a simple UI (flask) using Azure App Service to allow user choose files to upload, and then want to upload these files to the blob container.

My original design is UI -> Blob Container by Python Storage SDK:

containerClient.upload_blob(filename, file)

But I am facing the timeout issue due to Azure App Service when uploading large files.

So I change the upload UI with dropzone.js, and enable uploading in chunk, so that the server can consistently receive response to prevent timeout.

And another issue coming up is that upload process is executed for every piece of chunk, and blob container only receives the last chunk of the data that I upload. (From the document, I know that the chunking is automatically used in blob upload, I wonder if we are able to track the progress of the upload??? if so, I probably don't need to use dropzone.js for uploading in chunk).

I also tried another approach by creating Azure App Function (HTTPS trigger), and then send an http trigger to that endpoint to start the blob upload.

for file in files:
    fileToSend = {'file': (f.filename, f.stream, f.content_type, f.headers)}
    r = requests.post('https://myazurefunctionapp.azurewebsites.net/api/funcName', files=fileToSend)

In the azure function, I use Python Storage SDK to connect to container and then upload blob

container = ContainerClient.from_connection_string(conn_str, container_name)
for k, f in req.files.items():
    container.upload_blob(f.filename, f)

But I notice that the function is triggered by piece of chunk (request), and I also end up with only receiving the last chunk of data in the container.

I wonder what would be the better workflow? or if there any way that makes sure the upload is completed (in azure function) and then start the upload to blob container.

Many Thanks,

Upvotes: 0

Views: 1118

Answers (1)

Kartik Bhiwapurkar
Kartik Bhiwapurkar

Reputation: 5159

• Storage clients default to a 32 MB maximum single block upload. When a block blob upload is larger than the value in ‘SingleBlobUploadThresholdInBytes’ property, storage clients break the file into blocks of maximum allowed size and try to upload it. Since the block blob size that you are trying to upload is greater than 32 MB, it throws an exception and breaks the file into allowed smaller chunks. Also, you might not be using the correct ‘Blob service client’ which interacts with the resources, i.e., storage account, blob storage containers and blobs.

Below is an example of the code for client object creation which requires a storage account’s blob service account URL and a credential that allows you to access a storage account: -

 from azure.storage.blob import BlobServiceClient
 service = BlobServiceClient(account_url="https://<my-storage-account-name>.blob.core.windows.net/", credential=credential)

• Thus, similarly, as you are using the above code in python to create a blob service client for interacting with storage accounts, kindly refer to the below documentation link that describes in detail as in how to develop a python code to integrate it with blob storage for storing massive amounts of unstructured data, such as text or binary data.

https://learn.microsoft.com/en-us/python/api/overview/azure/storage-blob-readme?view=azure-python

You can deploy this code in your app service or function and set the trigger accordingly for uploading and downloading blobs from the storage account. It also describes as in how you can configure authentication for this process to ensure that the correct user and files are being given access.

And refer to the documentation link for details on how to configure a blob trigger function in Azure for various interactions with the storage account when any users initiate any transaction through it.

https://learn.microsoft.com/en-us/azure/storage/blobs/blob-upload-function-trigger?tabs=azure-portal

Upvotes: 0

Related Questions