CODEWITHSUNDEEP
ajaxnode.jsazuresails.jsazure-web-app-service
Megakoresh
Megakoresh

Reputation: 824

How to pipe upload stream of large files to Azure Blob storage via Node.js app on Azure Websites?

I am building a Video service using Azure Media Services and Node.js Everything went semi-fine till now, however when I tried to deploy the app to Azure Web Apps for hosting, any large files fail with 404.13 error.

Yes I know about maxAllowedContentLength and not, that is NOT a solution. It only goes up to 4GB, which is pathetic - even HDR environment maps can easily exceed that amount these days. I need to enable users to upload files up to 150GB in size. However when azure web apps recieves a multipart request it appears to buffer it into memory until a certain threshold of either bytes or just seconds (upon hitting which, it returns me a 404.13 or a 502 if my connection is slow) BEFORE running any of my server logic.

I tried Transfer-Encoding: chunked header in the server code, but even if that would help, since Web Apps doesn't let the code run, that doesn't actually matter.

For the record: I am using Sails.js at backend and Skipper is handling the stream piping to Azure Blob Service. Localhost obviously works just fine regardless of file size. I made a duplicate of this question on MSDN forums, but those are as slow as always. You can go there to see what I have found so far: enter link description here

Clientside I am using Ajax FormData to serialize the fields (one text field and one file) and send them, using the progress even to track upload progress.

Is there ANY way to make this work? I just want it to let my serverside logic handle the data stream, without buffering the bloody thing.

Upvotes: 2

Views: 2447

Answers (1)

Rick Rainey
Rick Rainey

Reputation: 11246

Rather than running all this data through your web application, you would be better off having your clients upload directly to a container in your Azure blob storage account.

You will need to enable CORS on your Azure Storage account to support this. Then, in your web application, when a user needs to upload data you would instead generate a SAS token for the storage account container you want the client to upload to and return that to the client. The client would then use the SAS token to upload the file into your storage account.

On the back-end, you could fire off a web job to do whatever processing you need to do on the file after it's been uploaded.

Further details and sample ajax code to do this is available in this blog post from the Azure Storage team.

Upvotes: 4

Related Questions