Cloudfront - item in query string triggers 403 Forbidden from S3 origin on the same resource

Question

I have been debugging an issue with Cloudfront whereby specifically including the term "notification" in the query string of my request URL returns a 403 from S3. The two following examples exhibit this behaviour:

https://my.website.com/app/home?notification=THING - 403 Forbidden returned by S3

https://my.website.com/app/home?notificatio=THING - 304 Not Modified, and I can access the resource(s) without issue

To the extent of my testing, any content in the query string that explicitly does not include the string "notification" works. It's just this word in particular which is problematic.

Some things to note...

• Query String Forwarding and Caching on the origin behaviour is set to "Forward all, cache based on all"
• I have a Lambda function triggering on origin-request, origin-response & viewer-response events; this function appends /index.html to the request URI before forwarding to the origin.
• Both requests are using the same Origin Access Identity which I can verify has adequate permissions to S3 - this is proven by the second example working as expected.

As I understand it, S3 pays no attention to forwarded query strings, however extracting the request/response data from both of the examples above clearly shows a difference in behaviour, despite the only variable being the contents of the query string (that I can tell anyway!)

Failed request:

{
    "config": {
        "distributionDomainName": "<redacted>.cloudfront.net",
        "distributionId": "<redacted>",
        "eventType": "origin-response",
        "requestId": "<redacted>"
    },
    "request": {
        "clientIp": "<redacted>",
        "headers": {
            "x-forwarded-for": [
                {
                    "key": "X-Forwarded-For",
                    "value": "<redacted>"
                }
            ],
            "user-agent": [
                {
                    "key": "User-Agent",
                    "value": "Amazon CloudFront"
                }
            ],
            "via": [
                {
                    "key": "Via",
                    "value": "1.1 <redacted>.cloudfront.net (CloudFront)"
                }
            ],
            "host": [
                {
                    "key": "Host",
                    "value": "<redacted>.s3.eu-west-1.amazonaws.com"
                }
            ]
        },
        "method": "GET",
        "origin": {
            "s3": {
                "authMethod": "origin-access-identity",
                "customHeaders": {},
                "domainName": "<redacted>.s3.eu-west-1.amazonaws.com",
                "path": "",
                "region": "eu-west-1"
            }
        },
        "querystring": "notification=THING",
        "uri": "/app/home/index.html"
    },
    "response": {
        "headers": {
            "x-amz-request-id": [
                {
                    "key": "x-amz-request-id",
                    "value": "<redacted>"
                }
            ],
            "x-amz-id-2": [
                {
                    "key": "x-amz-id-2",
                    "value": "<redacted>"
                }
            ],
            "date": [
                {
                    "key": "Date",
                    "value": "Tue, 15 Dec 2020 17:15:11 GMT"
                }
            ],
            "server": [
                {
                    "key": "Server",
                    "value": "AmazonS3"
                }
            ],
            "content-type": [
                {
                    "key": "Content-Type",
                    "value": "application/xml"
                }
            ],
            "transfer-encoding": [
                {
                    "key": "Transfer-Encoding",
                    "value": "chunked"
                }
            ]
        },
        "status": "403",
        "statusDescription": "Forbidden"
    }
}

Working:

{
    "config": {
        "distributionDomainName": "<redacted>.cloudfront.net",
        "distributionId": "<redacted>",
        "eventType": "origin-response",
        "requestId": "<redacted>"
    },
    "request": {
        "clientIp": "<redacted>",
        "headers": {
            "if-modified-since": [
                {
                    "key": "If-Modified-Since",
                    "value": "<redacted>"
                }
            ],
            "if-none-match": [
                {
                    "key": "If-None-Match",
                    "value": "<redacted>"
                }
            ],
            "x-forwarded-for": [
                {
                    "key": "X-Forwarded-For",
                    "value": "<redacted>"
                }
            ],
            "user-agent": [
                {
                    "key": "User-Agent",
                    "value": "Amazon CloudFront"
                }
            ],
            "via": [
                {
                    "key": "Via",
                    "value": "1.1 <redacted>.cloudfront.net (CloudFront)"
                }
            ],
            "host": [
                {
                    "key": "Host",
                    "value": "<redacted>.s3.eu-west-1.amazonaws.com"
                }
            ]
        },
        "method": "GET",
        "origin": {
            "s3": {
                "authMethod": "origin-access-identity",
                "customHeaders": {},
                "domainName": "<redacted>.s3.eu-west-1.amazonaws.com",
                "path": "",
                "region": "eu-west-1"
            }
        },
        "querystring": "notificatio=THING",
        "uri": "/app/home/index.html"
    },
    "response": {
        "headers": {
            "x-amz-id-2": [
                {
                    "key": "x-amz-id-2",
                    "value": "<redacted>"
                }
            ],
            "x-amz-request-id": [
                {
                    "key": "x-amz-request-id",
                    "value": "<redacted>"
                }
            ],
            "date": [
                {
                    "key": "Date",
                    "value": "Tue, 15 Dec 2020 17:17:29 GMT"
                }
            ],
            "last-modified": [
                {
                    "key": "Last-Modified",
                    "value": "Thu, 03 Dec 2020 09:14:18 GMT"
                }
            ],
            "etag": [
                {
                    "key": "ETag",
                    "value": "<redacted>"
                }
            ],
            "cache-control": [
                {
                    "key": "Cache-Control",
                    "value": "no-cache"
                }
            ],
            "server": [
                {
                    "key": "Server",
                    "value": "AmazonS3"
                }
            ]
        },
        "status": "304",
        "statusDescription": "Not Modified"
    }
}

Answer 1

Resurrecting this because I found it extremally hard to google this issue. First I narrowed it down to the problem being on the S3 side. I had a similar problem with "location" in my query string. Apparently both of these are being interpreted by the S3 API: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html#bucket-config-options-intro

I contacted support to see if there's a way to work around this and I got this:

"I was able to confirm using the log from the request ID you provided that these query strings are being interpreted as API calls. https://xxxx.s3.eu-west-1.amazonaws.com/index.html?location=test is being interpreted as a GetBucketLocation API operation. The notification query string would be interpreted as a GetBucketNotificationConfiguration operation. This being the case you will not be able to use these query string parameters such as 'location'. After consulting with some colleagues on the issue, I can confirm that this is as designed and cannot be changed. The recommendation here would be to change the query string used to something like 'loc' so that you don't send an unintentional API operation to S3."