AWS API authentication with Rest API - AWS was not able to validate the provided access credentials

Question

I'm trying to create a simple C# or VB program to connect to the AWS API and return a JSON string giving EC2 instance data. Instead I get the error "AWS was not able to validate the provided access credentials"

I've tried connecting in both Postman and in a VB program as follows:

Imports System.Net
Imports System.IO

Module Main

    Sub Main()

        Dim result As String = RunQuery("", "https://ec2.amazonaws.com/?Action=DescribeInstances")

        Console.WriteLine(result)
        Console.ReadLine()

    End Sub


    Public Function RunQuery(creds As String, url As String, Optional data As String = Nothing) As String

        Dim request As HttpWebRequest = TryCast(WebRequest.Create(url), HttpWebRequest)
        request.ContentType = "application/json"
        request.Method = "GET"

        If data IsNot Nothing Then
            Using writer As StreamWriter = New StreamWriter(request.GetRequestStream())
                writer.Write(data)
            End Using
        End If

        Dim d As Date = Date.UtcNow
        Dim t As String = Format(d, "yyyyMMdd") & "T" & Format(d, "HHmmss") & "Z"

        request.Headers.Add("X-Amz-Date", t)
        request.Headers.Add("Authorization", "AWS4-HMAC-SHA256 Credential=***MYAPIKEY***/20201216/us-east-2/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=***MYAPISECRET***")

        Dim response As HttpWebResponse = TryCast(request.GetResponse(), HttpWebResponse)
        Dim result As String = String.Empty

        Using reader As StreamReader = New StreamReader(response.GetResponseStream())
            result = reader.ReadToEnd()
        End Using

        Return result

    End Function
End Module

The output from postman was

<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>37153d9f-c042-4ae3-8a13-06dc3b052afc</RequestID></Response>

I wrote the vb based on postmans code:

var client = new RestClient("https://ec2.amazonaws.com/?Action=DescribeInstances");
client.Timeout = -1;
var request = new RestRequest(Method.GET);
request.AddHeader("X-Amz-Date", "20201216T092737Z");
request.AddHeader("Authorization", "AWS4-HMAC-SHA256 Credential=MYAPIKEY/20201216/us-east-2/ec2/aws4_request, SignedHeaders=host;x-amz-date, Signature=MYAPISIGNATURE");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);

Similar questions have been asked many times before but the only answers I can see are that the PC clock was wrong, so I've corrected mine such that its within 0.2 seconds of the true internet time.

Also I could not find another similarly simple VB or C# program that successfully connects to EC2, so this thread may help others in future.

I'm wondering if the signature has to be encoded with base64 or some other algorithm ? Otherwise how can I get this working either in postman or in vb/c# ? TIA

Answer 1

So there were a couple of issues

1. Wrong URL, the correct one was https://ec2.us-east-2.amazonaws.com/?Action=DescribeInstances&Version=2016-11-15

2. I assumed that the secret was the signature, it is not.

Here's a complete VB.NET console application that successfully connects to AWS. Sorry its not commented but the code is reasonably obvious and not too long.

Option Explicit On
Option Compare Text
Option Strict On

Imports System.Net.Http
Imports System.Security.Cryptography
Imports System.Text

Public Module Program

    Public Sub Main(ByVal args As String())

        If args.Length <> 5 Then
            Throw New Exception("AWS Integration requires 5 parameters: url accessKey secretKey awsRegion awsServiceName")
        End If

        Dim url As String = args(0)             '   "https://ec2.us-east-2.amazonaws.com/?Action=DescribeInstances&Version=2016-11-15"
        Dim accessKey As String = args(1)       ' api key
        Dim secretkey As String = args(2)       ' api secret
        Dim awsRegion As String = args(3)       ' = "us-east-2"
        Dim awsServiceName As String = args(4)  '= "ec2"

        Dim msg As HttpRequestMessage = New HttpRequestMessage(HttpMethod.[Get], url)
        msg.Headers.Host = msg.RequestUri.Host
        Dim utcNowSaved As DateTimeOffset = DateTimeOffset.UtcNow
        Dim amzLongDate As String = utcNowSaved.ToString("yyyyMMddTHHmmssZ")
        Dim amzShortDate As String = utcNowSaved.ToString("yyyyMMdd")
        msg.Headers.Add("x-amz-date", amzLongDate)
        Dim canonicalRequest As New StringBuilder
        canonicalRequest.Append(msg.Method.ToString & vbLf)
        canonicalRequest.Append(String.Join("/", msg.RequestUri.AbsolutePath.Split("/"c).Select(AddressOf Uri.EscapeDataString)) & vbLf)
        canonicalRequest.Append(GetCanonicalQueryParams(msg) & vbLf)
        Dim headersToBeSigned As New List(Of String)

        For Each header In msg.Headers.OrderBy(Function(a) a.Key.ToLowerInvariant, StringComparer.OrdinalIgnoreCase)
            canonicalRequest.Append(header.Key.ToLowerInvariant)
            canonicalRequest.Append(":")
            canonicalRequest.Append(String.Join(",", header.Value.[Select](Function(s) s.Trim)))
            canonicalRequest.Append(vbLf)
            headersToBeSigned.Add(header.Key.ToLowerInvariant)
        Next

        canonicalRequest.Append(vbLf)
        Dim signedHeaders As String = String.Join(";", headersToBeSigned)
        canonicalRequest.Append(signedHeaders & vbLf)
        canonicalRequest.Append("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
        Dim stringToSign As String = "AWS4-HMAC-SHA256" & vbLf & amzLongDate & vbLf & amzShortDate & "/" & awsRegion & "/" & awsServiceName & "/aws4_request" & vbLf & Hash(Encoding.UTF8.GetBytes(canonicalRequest.ToString))
        Dim dateKey() As Byte = HmacSha256(Encoding.UTF8.GetBytes("AWS4" & secretkey), amzShortDate)
        Dim dateRegionKey() As Byte = HmacSha256(dateKey, awsRegion)
        Dim dateRegionServiceKey() As Byte = HmacSha256(dateRegionKey, awsServiceName)
        Dim signingKey() As Byte = HmacSha256(dateRegionServiceKey, "aws4_request")
        Dim signature As String = ToHexString(HmacSha256(signingKey, stringToSign.ToString))
        Dim credentialScope As String = amzShortDate & "/" & awsRegion & "/" & awsServiceName & "/aws4_request"
        msg.Headers.TryAddWithoutValidation("Authorization", "AWS4-HMAC-SHA256 Credential=" & accessKey & "/" & credentialScope & ", SignedHeaders=" & signedHeaders & ", Signature=" & signature)
        Dim client As HttpClient = New HttpClient
        Dim result As HttpResponseMessage = client.SendAsync(msg).Result

        If result.IsSuccessStatusCode Then
            'Console.WriteLine(result.Headers)
            Dim s As String = result.Content.ReadAsStringAsync().Result
            Console.WriteLine(s)
        Else
            Console.WriteLine(result.StatusCode & vbCrLf & result.ToString.Replace(vbCr, "").Replace(vbLf, ""))
        End If

    End Sub

    Private Function GetCanonicalQueryParams(ByVal request As HttpRequestMessage) As String
        Dim values = New SortedDictionary(Of String, String)
        Dim querystring = System.Web.HttpUtility.ParseQueryString(request.RequestUri.Query)

        For Each key In querystring.AllKeys

            If key Is Nothing Then
                values.Add(Uri.EscapeDataString(querystring(key)), $"{Uri.EscapeDataString(querystring(key))}=")
            Else
                values.Add(Uri.EscapeDataString(key), $"{Uri.EscapeDataString(key)}={Uri.EscapeDataString(querystring(key))}")
            End If
        Next

        Dim queryParams = values.Select(Function(a) a.Value)
        Return String.Join("&", queryParams)
    End Function

    Public Function Hash(ByVal bytesToHash() As Byte) As String
        Return ToHexString(SHA256.Create.ComputeHash(bytesToHash))
    End Function

    Private Function ToHexString(ByVal array As IReadOnlyCollection(Of Byte)) As String
        Dim hex = New StringBuilder(array.Count * 2)

        For Each b In array
            hex.AppendFormat("{0:x2}", b)
        Next

        Return hex.ToString
    End Function

    Private Function HmacSha256(ByVal key() As Byte, ByVal data As String) As Byte()
        Return New HMACSHA256(key).ComputeHash(Encoding.UTF8.GetBytes(data))
    End Function
End Module

HTH someone :)