Nginx reverse proxy to Heroku fails SSL handshake

Question

I'm unfortunately not much of a system administrator and have come upon a problem that has me banging my head against the wall.

The short story is that I'm running Nginx on EC2 (Ubuntu 14.04.4 LTS) to (a) host my company's marketing site (https://example.com, which incidentally is Wordpress) and (b) serve as a reverse proxy to our Rails app running on Heroku (https:// app.example.com), for certain paths. We use the same SSL certificate for both example.com and app.example.com. All of this has worked great for 8-10 months, but I recently switched from Heroku's paid SSL addon to the new free SSL offering, and now our reverse proxy is broken.

In checking the Nginx error logs, I see the following:

SSL_do_handshake() failed (SSL: error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: ipaddress1, server: example.com, request: "GET /proxiedpath/proxiedpage HTTP/1.1", upstream: "https:// ipaddress2:443/proxiedpath/proxiedpage", host: "example.com"

I've tried to search around for some additional guidance - I've upgraded Nginx (1.10.1) and OpenSSL (1.0.2h) with no luck. I suspected the issue might be due to Heroku's use of SNI in the new free SSL feature (https://devcenter.heroku.com/articles/ssl-beta), but haven't been able to determine why this might be a problem.

A few additional points on my exploration to this point:

• When I switched to the new free Heroku SSL, I changed our app.example.com DNS record to point to app.example.com.herokudns.com, as instructed by the docs. The application can be accessed normally through app.example.com and when I run an nslookup on app.example.com and app.example.com.herokudns.com, I get the same IP address back. However...

• I cannot access the application through either the IP address returned from nslookup or app.example.com.herokudns.com. I suspect this is normal and expected but don't know enough to say exactly why this is. And...

• The IP address returned from the nslookup is NOT the same as the IP address referenced in the log's error message above ("ipaddress2"). In fact, "ipaddress2" is not consistent throughout the logs - it seems to change regularly. Again I don't know enough to know what I don't know... load balancing on Heroku's side?

And finally, my Nginx reverse proxy is configured as follows in nginx.conf:

http {

    client_max_body_size 500M;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    server_names_hash_bucket_size 64;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    gzip on;
    gzip_disable "msie6";

    server {

        listen 443 default_server;
        server_name example.com;

        root /usr/share/nginx/html;
        index index.php index.html index.htm;

        ssl on;
        ssl_certificate mycompanycert.crt;
        ssl_certificate_key mycompanykey.key;

        ssl_session_timeout 5m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
        ssl_prefer_server_ciphers on;

        error_page 404 /404.html;
        error_page 500 502 503 504 /50x.html;

        location / {
            try_files $uri $uri/ /index.php?q=$uri&$args;
        }

        location ^~ /proxiedpath/ {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-Proto https;
            proxy_pass https://app.example.com/proxiedpath/;
        }

    }

}

Any help is greatly appreciated - thanks very much!

Answer 1

In case someone experience the same issue and above directives are not enough, you will need to use "proxy_ssl_name name" - https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_name:

proxy_ssl_server_name on; #mandatory

proxy_ssl_protocols TLSv1.3; #sometimes is needed

proxy_ssl_name myfqdn.com;

Why? Sometimes proxy_set_header Host myfqdn.com; is not enough to satisfy upstream group server's SNI.

Answer 2

Just wanted to chime in and say after days of banging my head on my desk, I finally got my issue figured out. Wanted to post it here in case it helps others.

My issue was that I only had the server cert and key configured, and after running a test on https://www.ssllabs.com, it reported that the cert chain was incomplete. I changed the server cert to be the complete chain, and my issue is now resolved.

Hope that helps someone else in the same boat.

Answer 3

Please try by adding proxy_ssl_server_name on

location ^~ /proxiedpath/ {
    proxy_ssl_server_name on;
 }

Answer 4

As a note for others a related condition that Heroku imposes is that the HOST field must match the custom domain name.

So in addition to proxy_ssl_server_name you may also want to set a line like:

proxy_set_header Host mycustomdomain.com;

Of course this only applies if the host field incoming into the sever is different from the domain that your server resides in.

The specific error you get is:

SSL certificate error

There is conflicting information between the SSL connection, its certificate and/or the included HTTP requests.

Answer 5

I was able to solve this today and wanted to post the solution in case others run into the same issue.

It turns out that the problem was related to SNI after all. I found this ticket on nginx.org:

https://trac.nginx.org/nginx/ticket/229

Which led me to the proxy_ssl_server_name directive:

http://nginx.org/r/proxy_ssl_server_name

By setting to "on" in your config, you'll be able to proxy to upstream hosts using SNI.

Thanks to all who commented with suggestions!