CODEWITHSUNDEEP
c#azure-functionsazure-blob-storage
STM
STM

Reputation: 23

How to stream binary file from the Blob Storage through Azure Function

I'm trying to get a binary file from the Blob Storage through Azure API management by using Azure function with HTTP trigger. How the Azure Function should be implemented to reach smallest possible memory footprint?

This implementation works but it seems to require a lot of memory which leads to the out of memory exception when too many concurrent request are processed:

public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "dbrevisions/{dbRevision}")] HttpRequestMessage request,
    [Blob("typedatadev/typedata_{dbRevision}.db", FileAccess.Read)] Stream blobStream,
    string dbRevision,
    ILogger log)
{
    var memoryBlobStream = new MemoryStream();
    blobStream.CopyTo(memoryBlobStream);

    var result = new FileStreamResult(memoryBlobStream, "application/octet-stream");
    result.FileStream.Flush();
    result.FileStream.Seek(0, SeekOrigin.Begin);
    return result;
}

This will end up to the status 502 (also without the "Flush" call):

public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = "dbrevisions/{dbRevision}")] HttpRequestMessage request,
    [Blob("typedatadev/typedata_{dbRevision}.db", FileAccess.Read)] Stream blobStream,
    string dbRevision, ILogger log)
{
    var result = new FileStreamResult(blobStream, "application/octet-stream");
    result.FileStream.Flush();
    result.FileStream.Seek(0, SeekOrigin.Begin);
    return result;
}

This also fails:

var response = new HttpResponseMessage(HttpStatusCode.OK)
{
    Content = new StreamContent(blobStream)
};
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
return response;

Upvotes: 2

Views: 7051

Answers (2)

Rakesh
Rakesh

Reputation: 117

The question was asked in 2020, I came across the post looking for the same solution for my code which is on .net6 VisualStudio 2022. Hence the latest libraries.

Now I am trying to read a pfx certificate file which is stored as a blob on an Azure Container. This container and blob access is Private.

I am connecting to the Azure Storage account, then finding the file, then fetchign the file as a memory stream from azure and finally returning that as a X509Certificate2 certificate object.

This is my final code, tested ok.

using Azure.Storage.Blobs;
using Microsoft.Extensions.Logging;
using System;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using System.Threading.Tasks;

namespace ConfigEncryption
{
    internal static class CloudStorage
    {
            //Connection string of the Azure Storage Account - Fonud at the storageaccount|Access keys - Show the keys
            const string connectionString = "DefaultEndpointsProtocol=https;AccountName=storageaccountname;AccountKey=thelongkeyvaluewithfewslahesinit;EndpointSuffix=core.windows.net";
            const string containerName = "yourcontainername";
            const string certName = "Your_Certificate.pfx";const string password = "the_certificate_password";
        
    public static async Task<X509Certificate2> GetCertAsync(ILogger log)
    {
        try
        {
            log.LogInformation("Connecting to cloud storage to fetch the certificate");
            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
            BlobClient blobClient = containerClient.GetBlobClient(certName);

            byte[] certData; 

            if (await blobClient.ExistsAsync())
            {
                var memorystream = new MemoryStream();
                blobClient.DownloadTo(memorystream);

                certData = memorystream.ToArray();

                X509Certificate2 cert = new X509Certificate2(certData, password);

                log.LogInformation("Found the certificate on cloud storage.");
                return cert;
            }

            log.LogError("Error: Cerificate not found in the container");
        }
        catch(Exception ex)
        {
            log.LogError($"Error: Getting certificate from Cloud storage. Exception Details :{ex.Message}");
        }
        return null;
    }
}

}

I hope it helps someone, happy coding !

Regards,

Rakesh

Upvotes: 0

krishg
krishg

Reputation: 6508

I did like below for minimum memory footprint. Note that instead of binding to stream, I am binding to a ICloudBlob instance (luckily, C# function supports several flavors of blob input binding) and returning open stream. Tested it using memory profiler and works fine with no memory leak even for large blobs.

NOTE: You don't need to seek to stream position 0 or flush or dispose (disposing would be automatically done on response end);

using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.Storage.Blob;

namespace TestFunction1
{
   public static class MyFunction
   {
        [FunctionName("MyFunction")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "dbrevisions/{dbRevision}")] HttpRequest req,
            [Blob("typedatadev/typedata_{dbRevision}.db", FileAccess.Read, Connection = "BlobConnection")] ICloudBlob blob,
            string dbRevision,
            ILogger log)
        {
            var blobStream = await blob.OpenReadAsync().ConfigureAwait(false);
            return new FileStreamResult(blobStream, "application/octet-stream");
        }
   }
}

Upvotes: 2

Related Questions