Nginx proxy_pass websocket timing out despite read_timeout being set

Question

So we need to proxy any traffic going to /notifications in nginx to an AWS ALB.

If we create a websocket directly to the ALB it closes after 3:10 seconds, which is what we have configured the timeout to be.

However, when we try from the front end proxy pass, we can't get a greater timeout than 60s. I have followed almost every thread I can find on the matter, so here is our config:

    location /notifications/internal/ {
        rewrite ^/notifications/internal /notinternal last;
    }

    location /notifications {
        rewrite ^/notifications(/|$)(.*) /$2 break;
        proxy_pass https://ALB;
        proxy_pass_request_headers on;
        proxy_read_timeout 3600s;
        proxy_connect_timeout 3600s;
        proxy_send_timeout 3600s;
        send_timeout 3600s;

        proxy_socket_keepalive on;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

After 60s exactly we always see this log:

10.31.0.22 - - [14/May/2020:13:15:16 +0000] "GET /notifications/ws HTTP/1.1" response 101 body-bytes 100 upstream_resp_time 60.002 req_time 60.001 req_bytes 373 vhost x.x country - cf_connecting_ip - cf_ray - 

I have tried adding greater timeouts in nginx.conf but nothing seems to stick.

I tried putting the proxy pass into an upstream block, but it doesn't seem to work, I get 404 not found.

Now I know nginx is playing some part in this as if I set the timeout to 5s, I do get the early timeout, something is not allowing it to be greater than 1minute.

Answer 1

The issue here is that our frontend was also behind an ALB with a timeout set to 60s. I feel so silly.