Azure Blob Storage Authorization

Question

EDIT 2: The following is the output of the Authorization error:

<?xml version=\"1.0\" encoding=\"utf-8\"?>
<Error>
  <Code>AuthenticationFailed</Code>
  <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:34d738a5-101e-000d-5a14-ed5956000000\nTime:2021-01-17T21:07:38.6231913Z
  </Message>
  <AuthenticationErrorDetail>Signature did not match. String to sign used was cw\n2021-01-17T19:06:42Z\n2021-01-18T21:06:42Z\n/blob/example-account/example-container/example-blob.json\n\n\nhttps\n2019-02-02\nb\n\n\n\n\n\n
  </AuthenticationErrorDetail>
</Error>

I don't really understand... I updated the C# code below to print out the string_to_sign with the \n characters, and it's exactly the same as the string_to_sign from the output above.

NOTE: The SAS token generated from Azure Storage that does work is an Account SAS, while the one I'm generating is a Service SAS. Could Service SAS be restricted in Azure Storage?

EDIT: I tried generating a SAS token directly from Azure Storage, and this did seem to work. It appears to be an account SAS, NOT the service SAS I'm trying to use below.

?sv=2019-12-12&ss=b&srt=o&sp=wac&se=2021-01-18T01:15:13Z&st=2021-01-17T17:15:13Z&spr=https&sig=<signature>

I'm looking to be able to send upload a file to Azure Storage using it's REST API. However, I'm having some trouble getting it to Authorize. I find the documentation a bit conflicting, in some places it says I can include a SAS token in the URI, in others it's in the Authorize header. For context, I'm trying to do it directly from APIM, so in the sample code below it's written with its limited API. This is just a general concept I'm using to generate the authorization string, but I keep getting a 403 when I use it (I'm not sure if I need to do something from the Azure Storage side).

/**
 Based on https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas
*/
using System;
using System.Collections.Generic;

namespace sas_token
{
  class Program
  {
    static void Main(string[] args)
    {
      string key = args[0];
      Console.WriteLine(generate_blob_sas_token(key));
    }

    public static string generate_blob_sas_token(string key)
    {
      const string canonicalizedResource = "canonicalizedResource";

      // NOTE: this only works for Blob type files, Tables have a different
      // structure
      // NOTE: use a List instead of Dictionary since the order of keys in
      // Dictionaries is undefined and the signature string requires a very
      // specific order
      List<KeyValuePair<string, string>> sas_token_properties = new List<KeyValuePair<string, string>>(){


    // signedPermissions, select 1..* from [racwdxltmeop], MUST be in that order
    new KeyValuePair<string, string>("sp", "cw"),

    // signedStart time, date from when the token is valid
    // NOTE: because of clock skew between services, even setting the time to
    // now may not create an immediately usable token
    new KeyValuePair<string, string>("st", DateTime.UtcNow.AddMinutes(-120).ToString("yyyy-MM-ddTHH:mm:ssZ")),

    // signedExpiry time, date until the token is valid
    new KeyValuePair<string, string>("se", DateTime.UtcNow.AddDays(1).ToString("yyyy-MM-ddTHH:mm:ssZ")),

    // canonicalizedResource, must be prefixed with /blob in recent versions
    // NOTE: this is NOT included as a query parameter, but is in the signature
    // URL = https://myaccount.blob.core.windows.net/music/intro.mp3
    // canonicalizedResource = "/blob/myaccount/music/intro.mp3"
    new KeyValuePair<string, string>(canonicalizedResource, "/blob/example-account/example-container"),

    // signedIdentifier, can be used to identify a Stored Access Policy
    new KeyValuePair<string, string>("si", ""),

    // signedIP, single or range of allowed IP addresses
    new KeyValuePair<string, string>("sip", ""),

    // signedProtocol
    // [http, https]
    new KeyValuePair<string, string>("spr", "https"),

    // signedVersion, the version of SAS used (defines which keys are
    // required/available)
    new KeyValuePair<string, string>("sv", "2019-02-02"),

    // signedResource, the type of resource the token is allowed to access
    // [b = blob, d = directory, c = container, bv, bs]
    new KeyValuePair<string, string>("sr", "b"),

    // signedSnapshotTime
    new KeyValuePair<string, string>("sst", ""),


    // the following specify how the response should be formatted

    // Cache-Control
    new KeyValuePair<string, string>("rscc", ""),

    // Content-Disposition
    new KeyValuePair<string, string>("rscd", ""),

    // Content-Encoding
    new KeyValuePair<string, string>("rsce", ""),

    // Content-Language
    new KeyValuePair<string, string>("rscl", ""),

     // Content-Type
    new KeyValuePair<string, string>("rsct", "")
};

      // the format is a very specific text string, where values are delimited by new
      // lines, and the order of the properties in the string is important!
      List<string> values = new List<string>();
      foreach (KeyValuePair<string, string> entry in sas_token_properties)
      {
        values.Add(entry.Value);
      }
      string string_to_sign = string.Join("\n", new List<string>(values));
      Console.WriteLine(string_to_sign.Replace("\n", "\\n"));
      System.Security.Cryptography.HMACSHA256 hmac = new System.Security.Cryptography.HMACSHA256(System.Text.Encoding.UTF8.GetBytes(key));
      var signature = System.Convert.ToBase64String(hmac.ComputeHash(System.Text.Encoding.UTF8.GetBytes(string_to_sign)));

      // create the query parameters of any set values + the signature
      // NOTE: all properties that contribute to the signature must be added
      // as query params EXCEPT canonicalizedResource
      List<string> parameters = new List<string>();

      foreach (KeyValuePair<string, string> entry in sas_token_properties)
      {
        if (!string.IsNullOrEmpty(entry.Value) && entry.Key != canonicalizedResource)
        {
          parameters.Add(entry.Key + "=" + System.Net.WebUtility.UrlEncode(entry.Value));
        }
      }

      parameters.Add("sig=" + System.Net.WebUtility.UrlEncode(signature));

      string sas_token_querystring = string.Join("&", parameters);

      return sas_token_querystring;
    }
  }
}

I use the output in the following (simplified) APIM policy (I set "sas_token" variable to the output of the function to test the process):

<set-variable name="x-request-body" value="@(context.Request.Body.As<string>())" />
<send-request mode="new" response-variable-name="tokenstate" timeout="20" ignore-error="true">
   <set-url>@("https://example-account.blob.core.windows.net/example-container/test.json")</set-url>
   <set-method>PUT</set-method>
   <set-header name="x-ms-date" exists-action="override">
     <value>@(DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ"))</value>
   </set-header>
   <set-header name="x-ms-version" exists-action="override">
     <value>2019-02-02</value>
   </set-header>
   <set-header name="x-ms-blob-type" exists-action="override">
     <value>BlockBlob</value>
   </set-header>
   <set-header name="Authorization" exists-action="override">
     <value>@("SharedAccessSignature " + (string)context.Variables["sas_token"])</value>
   </set-header>
   <set-body>@((string)context.Variables["x-request-body"])</set-body>
</send-request>

For completeness, here's the result from APIM when I trace a test request using {"hello": "then"}:

{
    "message": "Request is being forwarded to the backend service. Timeout set to 20 seconds",
    "request": {
        "method": "PUT",
        "url": "https://example-account.blob.core.windows.net/example-container/test.json",
        "headers": [
            {
                "name": "Host",
                "value": "example-account.blob.core.windows.net"
            },
            {
                "name": "Content-Length",
                "value": 17
            },
            {
                "name": "x-ms-date",
                "value": "2021-01-17T16:53:28Z"
            },
            {
                "name": "x-ms-version",
                "value": "2019-02-02"
            },
            {
                "name": "x-ms-blob-type",
                "value": "BlockBlob"
            },
            {
                "name": "Authorization",
                "value": "SharedAccessSignature sp=cw&st=2021-01-17T13%3A42%3A02Z&se=2021-01-18T15%3A42%3A02Z&spr=https&sv=2019-02-02&sr=b&sig=<signature>"
            },
            {
                "name": "X-Forwarded-For",
                "value": "205.193.94.40"
            }
        ]
    }
}
send-request (92.315 ms)
{
    "response": {
        "status": {
            "code": 403,
            "reason": "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."
        },
        "headers": [
            {
                "name": "x-ms-request-id",
                "value": "185d86f5-601e-0038-5cf1-ec3542000000"
            },
            {
                "name": "Content-Length",
                "value": "321"
            },
            {
                "name": "Content-Type",
                "value": "application/xml"
            },
            {
                "name": "Date",
                "value": "Sun, 17 Jan 2021 16:53:28 GMT"
            },
            {
                "name": "Server",
                "value": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0"
            }
        ]
    }
}

Also, still newish to C#, so if something can be done better please let me know.

Answer 1

Thanks to David's help to confirm that it was my error, I was incorrectly converting the key to generate the HMAC. Below is the correct code, notice the Base64 decode, whereas originally I was just getting the byte array:

string string_to_sign = string.Join("\n", new List<string>(values));
Console.WriteLine(string_to_sign.Replace("\n", "\\n"));
System.Security.Cryptography.HMACSHA256 hmac = new System.Security.Cryptography.HMACSHA256(System.Convert.FromBase64String(key));
var signature = System.Convert.ToBase64String(hmac.ComputeHash(System.Text.Encoding.UTF8.GetBytes(string_to_sign)));

And then I can use it like so in the APIM Policy:

<set-variable name="x-request-body" value="@(context.Request.Body.As<string>())" />
<send-request mode="new" response-variable-name="tokenstate" timeout="20" ignore-error="true">
   <set-url>@(string.Format("https://example-account.blob.core.windows.net/example-container/test.json?{0}", context.Variables["sas_token"]))</set-url>
   <set-method>PUT</set-method>
   <set-header name="x-ms-date" exists-action="override">
     <value>@(DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ"))</value>
   </set-header>
   <set-header name="x-ms-version" exists-action="override">
     <value>2019-02-02</value>
   </set-header>
   <set-header name="x-ms-blob-type" exists-action="override">
     <value>BlockBlob</value>
   </set-header>
   <set-body>@((string)context.Variables["x-request-body"])</set-body>
</send-request>

Answer 2

Azure Storage supports below authorizing method:

But SAS token can not be the Authorization header of REST API.

https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-requests-to-azure-storage

I encapsulated several authentication methods:

using Azure.Storage;
using Azure.Storage.Sas;
using Microsoft.Azure.Services.AppAuthentication;
using System;
using System.IO;
using System.Net;
using System.Security.Cryptography;
using System.Text;

namespace ConsoleApp31
{
    class Program
    {
        static void Main(string[] args)
        {

            string storageKey = "xxxxxx";
            string storageAccount = "yourstorageaccountname";
            string containerName = "test";
            string blobName = "test.txt";
            string mimeType = "text/plain";

            string test = "This is a test of bowman.";
            byte[] byteArray = Encoding.UTF8.GetBytes(test);
            MemoryStream stream = new MemoryStream(byteArray);

            UseRestApiToUpload(storageKey,storageAccount,containerName,blobName,stream,mimeType);

            Console.WriteLine("*******");
            Console.ReadLine();
        }

        //Upload blob with REST API
        static void UseRestApiToUpload(string storageKey, string storageAccount, string containerName, string blobName, Stream stream, string mimeType)
        {
            string method = "PUT";
            long contentlength = stream.Length;

            string requestUri = $"https://{storageAccount}.blob.core.windows.net/{containerName}/{blobName}";

            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);

            string utcnow = DateTime.UtcNow.ToString("R");

            var memoryStream = new MemoryStream();
            stream.CopyTo(memoryStream);
            var content = memoryStream.ToArray();

            request.Method = method;

            request.Headers.Add("Content-Type", mimeType);
            request.Headers.Add("x-ms-version", "2019-12-12");
            request.Headers.Add("x-ms-date", utcnow);
            request.Headers.Add("x-ms-blob-type", "BlockBlob");
            request.Headers.Add("Content-Length", contentlength.ToString());

            //Use SharedKey to authorize.
            request.Headers.Add("Authorization", AuthorizationHeaderWithSharedKey(method, utcnow, request, storageAccount, storageKey, containerName, blobName));

            //Can not use SAS token in REST API header to authorize. 

            //Use Bearer token to authorize.
            //request.Headers.Add("Authorization",AuthorizationHeaderWithAzureActiveDirectory());



            using (Stream requestStream = request.GetRequestStream())
            {
                requestStream.Write(content, 0, (int)contentlength);
            }

            using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
            {
            }

        }

        //Use shared key to authorize.
        public static string AuthorizationHeaderWithSharedKey(string method, string now, HttpWebRequest request, string storageAccount, string storageKey, string containerName, string blobName)
        {

            string headerResource = $"x-ms-blob-type:BlockBlob\nx-ms-date:{now}\nx-ms-version:2019-12-12";
            string urlResource = $"/{storageAccount}/{containerName}/{blobName}";
            string stringToSign = $"{method}\n\n\n{request.ContentLength}\n\n{request.ContentType}\n\n\n\n\n\n\n{headerResource}\n{urlResource}";

            HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(storageKey));
            string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

            String SharedKey = String.Format("{0} {1}:{2}", "SharedKey", storageAccount, signature);
            return SharedKey;
        }


        //Use Shared access signature(SAS) to authorize.
        public static string AuthorizationHeaderWithSharedAccessSignature(string storageAccount, string storageKey)
        {
            // Create a SAS token that's valid for one hour.
            AccountSasBuilder sasBuilder = new AccountSasBuilder()
            {
                Services = AccountSasServices.Blobs | AccountSasServices.Files,
                ResourceTypes = AccountSasResourceTypes.Service,
                ExpiresOn = DateTimeOffset.UtcNow.AddHours(1),
                Protocol = SasProtocol.Https
            };

            sasBuilder.SetPermissions(AccountSasPermissions.Read |
                AccountSasPermissions.Write);

            // Use the key to get the SAS token.
            StorageSharedKeyCredential key = new StorageSharedKeyCredential(storageAccount, storageKey);
            string sasToken = sasBuilder.ToSasQueryParameters(key).ToString();

            Console.WriteLine("SAS token for the storage account is: {0}", sasToken);
            Console.WriteLine();

            return sasToken;
        }


        //Use Azure Active Directory(Bearer token) to authorize.
        public static string AuthorizationHeaderWithAzureActiveDirectory()
        {
            AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
            string bearertoken = azureServiceTokenProvider.GetAccessTokenAsync("https://storage.azure.com/").Result;
            return "Bearer " + bearertoken;
        }
    }
}

Although the interaction between many software packages and azure is based on REST API, for operations like uploading blobs, I don't recommend you to use rest api to complete. Azure officially provides many packaged packages that you can use directly, such as:

https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs?view=azure-dotnet

And example for .Net:

https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet

In the above SDK, you can use sas token for authentication.

Answer 3

I don't think you can put a SAS token in an Authorization header. I can't find any relevant sample, so I used the Using the Azure.Storage.Blob C# client library from NuGet to do this

var data = System.Text.Encoding.UTF8.GetBytes("Hello Azure Storage");

var keyCred = new StorageSharedKeyCredential(account, key);
var sasBuilder = new AccountSasBuilder()
{
    Services = AccountSasServices.Blobs,
    ResourceTypes = AccountSasResourceTypes.Object,
    ExpiresOn = DateTimeOffset.UtcNow.AddHours(1),
    Protocol = SasProtocol.Https
};
sasBuilder.SetPermissions(AccountSasPermissions.All);
var sasToken = sasBuilder.ToSasQueryParameters(keyCred).ToString();

var blobClient = new BlobServiceClient(new Uri($"https://{account}.blob.core.windows.net/?{sasToken}"), null);

var containter = blobClient.GetBlobContainerClient("test");
containter.UploadBlob("test.txt", new MemoryStream(data));

Generates an HTTP request like this:

PUT https://xxxxxx.blob.core.windows.net/test/test.txt?sv=2020-04-08&ss=b&srt=o&spr=https&se=2021-01-17T18%3A13%3A55Z&sp=rwdxlacuptf&sig=RI9It3O6mcmw********S%2B1r91%2Bj5zGbk%3D HTTP/1.1
Host: xxxxxx.blob.core.windows.net
x-ms-blob-type: BlockBlob
x-ms-version: 2020-04-08
If-None-Match: *
x-ms-client-request-id: c6e93312-af95-4a04-a207-2e2062b1dd26
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.8.0 (.NET Core 3.1.10; Microsoft Windows 10.0.19042)
Request-Id: |ffa2da23-45c79d128da40651.
Content-Length: 19

Hello Azure Storage

Then using the SAS token directly with WebClient,

var wc = new WebClient();
wc.Headers.Add("x-ms-blob-type: BlockBlob");
wc.UploadData($"https://{account}.blob.core.windows.net/test/test2.txt?{sasToken}", "PUT", data);

works too, which should be the minimal request:

PUT https://xxxxx.blob.core.windows.net/test/test2.txt?sv=2020-04-08&ss=b&srt=o&spr=https&se=2021-01-17T18%3A50%3A01Z&sp=rwdxlacuptf&sig=Fj4QVfwIfjXP10G%xxxxxxxx%2FF%2FcjikizKggY%3D HTTP/1.1
Host: xxxx.blob.core.windows.net
x-ms-blob-type: BlockBlob
Connection: Keep-Alive
Content-Length: 19

Hello Azure Storage

Removing the x-ms-blob-type header fails with:

The remote server returned an error: (400) An HTTP header that's mandatory for this request is not specified..

You are free to poke through the source code on GitHub for more details.