CODEWITHSUNDEEP
curlamazon-s3http-redirectdns
Marco Massenzio
Marco Massenzio

Reputation: 3012

DNS redirect to AWS S3 static website works with curl, but fails with browser

I own the covaxx.dev domain, but it is registered with NameCheap, not with AWS Route53.

I followed the basic guide to create a "static website" in S3, and this works just fine, index.html is served as expected, with the bucket named covaxx.dev (as it is supposed to).

On NameCheap, I have configured the DNS rule to enable URL Redirect, as described here using both the @ and www Host fields, with a type of URL Redirect Record and the Value the bucket's URL: http://covaxx.dev.s3-website-us-west-2.amazonaws.com/.

This pays off when I try to access the site using curl (or httpie for that matter):

curl -L www.covaxx.dev   
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Find Vaccine</title>
</head>
<body>
<h1>Find Vaccine</h1>

<p>This script checks CVS.com for appointment locations near you with available vaccine doses:
it uses the Google Map API to find those which are within a given distance
from the user's chosen location.</p>

<h6>Copyright &copy; 2021 AlertAvert.com  All rights reserved.</h6>
</body>
</html>

from the full trace[0] I can see the redirects point to the right place (both for www.covaxx.dev and covaxx.dev) - so far, so good.

However, when I tried to load the page in a browser (both Firefox, Linux and MacOS; and Safari, MacOS, iOS) they keep spinning and then time out with a "service timed out" error.

I'll be honest, this does not make any sense to me, and I suspect it may be something to do with my hyper-paranoid browser settings, but I'm not sure why is it happening at all. (I even disabled my VPN, in case it had an impact, but no change).

Any help in untangling the mystery would be mightily appreciated, thanks in advance!

[0] Full trace:

$ curl -v -L www.covaxx.dev
*   Trying 192.64.119.50:80...
* TCP_NODELAY set
* Connected to www.covaxx.dev (192.64.119.50) port 80 (#0)
> GET / HTTP/1.1
> Host: www.covaxx.dev
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx
< Date: Mon, 29 Mar 2021 00:12:07 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 76
< Connection: keep-alive
< Location: http://covaxx.dev.s3-website-us-west-2.amazonaws.com/
< X-Served-By: Namecheap URL Forward
< 
* Ignoring the response-body
* Connection #0 to host www.covaxx.dev left intact
* Issue another request to this URL: 'http://covaxx.dev.s3-website-us-west-2.amazonaws.com/'
*   Trying 52.218.234.242:80...
* TCP_NODELAY set
* Connected to covaxx.dev.s3-website-us-west-2.amazonaws.com (52.218.234.242) port 80 (#1)
> GET / HTTP/1.1
> Host: covaxx.dev.s3-website-us-west-2.amazonaws.com
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-amz-id-2: s6Hcwjf1E0Prv4QLJTGh9EaTZn5K1YTJy3FmQEtk598OIMyDv/wDeWFR/gzhgn1wgYAN5a/Y2bk=
< x-amz-request-id: 57BM0MN4P842GRFP
< Date: Mon, 29 Mar 2021 00:12:09 GMT
< Last-Modified: Mon, 29 Mar 2021 00:12:03 GMT
< ETag: "8853123d304bc0f26bbe8af3e7a1f71a"
< Content-Type: text/html
< Content-Length: 429
< Server: AmazonS3
< 
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Find Vaccine</title>
</head>
<body>
<h1>Find Vaccine</h1>

<p>This script checks CVS.com for appointment locations near you with available vaccine doses:
it uses the Google Map API to find those which are within a given distance
from the user's chosen location.</p>

<h6>Copyright &copy; 2021 AlertAvert.com  All rights reserved.</h6>
</body>
</html>
* Connection #1 to host covaxx.dev.s3-website-us-west-2.amazonaws.com left intact

Upvotes: 1

Views: 1126

Answers (1)

Marco Massenzio
Marco Massenzio

Reputation: 3012

Mystery solved, posting here in case anyone encounters the same issue.

The root cause is HSTS, a (relatively) new protocol that tells the browser to only use https with a certain domain - this wouldn’t normally matter, because we use http across the redirects (for now, anyway) and it should have worked just fine.

Alas, I was using URL Redirect with NameCheap, so the request was going to them, and probably there in the redirect maze, at some point the browser got told to switch to https. This setting is “sticky” in a browser, it gets cached (I’m told) for months, even.

Then, the AWS quirk is that, if you want to redirect a sub domain (e.g. www) you need to create an “redirect bucket”, empty, named exactly as the sub domain (www) and configured (in the Properties tab) to redirect to the original “domain bucket” (which, again, must be named exactly as the domain covaxx.dev).

Unfortunately, when setting up the bucket redirect, I misunderstood the instructions in the guide and instead of the covaxx.dev.s3-website-etc. I set it to s3://covaxx.dev.

So there were two errors, which confused the browser, but, amazingly, not curl (or httpie) and dig was happily resolving the domains.

To fix this I had to take two steps:

  1. obviously, fix the bucket redirect, pointing the www bucket to serve from covaxx.dev.s3-website-us-west-2.amazonaws.com;

  2. changed my DNS configuration at NameCheap to use a CNAME record, redirecting covaxx.dev to covaxx.dev.s3-website-us-west-2.amazonaws.com

Lo and behold, navigating to http://www.covaxx.dev now works (if you haven’t tried it before from your browser).

Upvotes: 3

Related Questions