CloudFront returns cached error response despite Cache-Control: no-cache header and minimum TTL of 0

Question

I am building a SPA with the frontend hosted by S3 and the backend served by Lambda and API Gateway. I am trying to use CloudFront to sit in front of both S3 and API Gateway in order to use the same domain name to avoid CORS issues and use cookies without the Domain attribute.

However, I am a CloudFront n00b and there is some caching behaviour in CloudFront that I do not understand.

For some reason, some of the API requests made from the client are receiving a 403 response, but that request is not making it to the Lambda -- the response seems to be a cached error response from CloudFront. The weirdest part is that the error response is not even the most recent response received from that (or any) URL. The response body for the 403 responses matches what my API server would send for unauthenticated requests sent to /api/*.

Here are the XHR requests that the client is making:

All of these API calls (including /api/activities/history/) have a cache-control header in the response that I thought would be preventing any of these responses from being cached at all:

cache-control: max-age=0, no-cache, no-store, must-revalidate, private

If it helps, the last response received by the client contains an x-cache header from CloudFront: x-cache: Error from cloudfront.

And here are the requests received by the origin:

[12/Jul/2020:19:44:16 +0000] "GET /api/user/ HTTP/1.1" 200 29 "" "Amazon CloudFront" 0/10.349
[12/Jul/2020:19:44:18 +0000] "GET /api/activities/history/ HTTP/1.1" 200 2036 "" "Amazon CloudFront" 0/1819.5900000000001
[12/Jul/2020:19:44:29 +0000] "GET /api/activities/in-progress/ HTTP/1.1" 200 139 "" "Amazon CloudFront" 0/3773.864
[12/Jul/2020:19:44:50 +0000] "GET /logout/ HTTP/1.1" 302 0 "" "Amazon CloudFront" 0/23.132
Forbidden: /api/user/
[12/Jul/2020:19:44:51 +0000] "GET /api/user/ HTTP/1.1" 403 58 "" "Amazon CloudFront" 0/1.839
[12/Jul/2020:19:45:12 +0000] "GET /login/github/ HTTP/1.1" 302 0 "" "Amazon CloudFront" 0/5.914
[12/Jul/2020:19:45:15 +0000] "GET /complete/github/ HTTP/1.1" 302 0 "" "Amazon CloudFront" 0/2289.144
[12/Jul/2020:19:45:16 +0000] "GET /api/user/ HTTP/1.1" 200 29 "" "Amazon CloudFront" 0/4.989999999999999
[12/Jul/2020:19:45:17 +0000] "GET /api/activities/in-progress/ HTTP/1.1" 200 139 "" "Amazon CloudFront" 0/714.226

You'll notice that the 403 response for /api/activities/history/ doesn't appear here. FYI the /logout/, /login/github/, /complete/github/ requests are from the same client, but are document requests from the client, which is why they don't appear in the screenshot above.

To add a little bit more detail to what is going on above, when the frontend loads it makes a request to the /api/user/ API endpoint. If the request is authenticated, then the frontend makes simultaneous requests to /api/activities/in-progress/ and /api/activities/history/. Otherwise, the user is presented with a login page. In the session shown above, I am able to render the main page fully, then I logout, load the main page and receive a 403 from /api/user/ rendering the login page. Then I click the login button, get redirected to Github, get redirected back, and then receive a 200 from /api/user/ generating simultaneous requests to /api/activities/in-progress/ and /api/activities/history/. The last API request is the one that fails, but shouldn't.

Here are the CloudFront behaviour settings for /api/*:

As you can see I am using the origin's cache headers.

I also created a custom error response for 403 responses to prevent caching of those responses:

Any ideas why I would be getting a cached 403 response for requests to /api/activities/history/ when the most recent response from that API endpoint was a 200?

Sorry for the long question, but thank you so much for reading!

EDIT: I've discovered that the simultaneous API requests seem to be central to this issue. If I change the frontend logic to make the requests serially then all requests reach the Lambda as expected.

Answer 1

Cloudfront caches error responses from the origin server by default for 10 seconds.
Copy/paste from the docs:

By default, CloudFront forwards error responses from the origin to the client. Additionally, CloudFront caches the origin's error response for 10 seconds by default.

If the error response from the origin contains a Cache-Control header, CloudFront caches the error with the relevant TTL instead of the default 5 minutes. CloudFront doesn't cache its own error responses, unless specified otherwise in a custom error response.

To determine whether the error response is from the origin or CloudFront, check the Server header. To determine whether the error is a cached response, check the Age header—a cached response includes an Age header.

Source: https://repost.aws/knowledge-center/cloudfront-custom-object-caching

If you look at the response headers of the cached request, you will notice a Age header with a value < 10 and with the X-cache header with value error from cloudfront.