CODEWITHSUNDEEP
varnishvarnish-vcl
bluethundr
bluethundr

Reputation: 1325

varnish 504 gateway timeout

I have 2 varnish servers located in digital ocean in the NY2 region. And I have two webservers sitting in Amazon EC2 us-east-1b.

Varnish is at version 3.05 and apache is running on the web servers at version 2.4.6. And what I've found is that every page that loads brings up a '504 Gateway Time-out' error, before loading the page normally. Sometimes if you hit the page a a second time it'll give the time out error again.

It's an annoying error that I'd like to try to correct. I tried addressing this by setting the timeouts to really high in the varnish config. That stopped the 503 errors I was getting, but now the 504 errors are the main problem.

I have a very simple varnish VCL file. Here it is:

backend web1 {
  .host = "10.10.10.25";
  .port = "80";
  .connect_timeout = 1200s;
  .first_byte_timeout = 1200s;
  .between_bytes_timeout = 1200s;
  .max_connections = 70;
  .probe = {
  .request =
   "GET /healthcheck.php HTTP/1.1"
   "Host: wiki.example.com"
   "Connection: close";
   .interval = 10m;
   .timeout = 60s;
   .window = 3;
   .threshold = 2;
   }
}

backend web2 {
  .host = "10.10.10.26";
  .port = "80";
  .connect_timeout = 1200s;
  .first_byte_timeout = 1200s;
  .between_bytes_timeout = 1200s;
  .max_connections = 70;
  .probe = {
  .request =
   "GET /healthcheck.php HTTP/1.1"
   "Host: wiki.example.com"
   "Connection: close";
   .interval = 10m;
   .timeout = 60s;
   .window = 3;
   .threshold = 2;
   }
}


director www round-robin {
  { .backend = web1;   }
  { .backend = web2;  }
 }


sub vcl_recv {

    if (req.url ~ "&action=submit($|/)") {
        return (pass);
    }

    set req.backend = www;
    return (lookup);
}

sub vcl_fetch {
      set beresp.ttl = 3600s;
      set beresp.grace = 4h;
      return (deliver);
}



sub vcl_deliver {
     if (obj.hits> 0) {
      set resp.http.X-Cache = "HIT";
     } else {
        set resp.http.X-Cache = "MISS";
     }
 }

And this is the error I'm getting in varnish log when a page gives the 504 error:

   10 TxHeader     c Age: 1
   10 TxHeader     c Via: 1.1 varnish
   10 TxHeader     c Connection: close
   10 TxHeader     c X-Cache: MISS
   10 Debug        c Write error, retval = -1, len = 75311, errno = Connection reset by peer
   10 ReqEnd       c 1481957120 1436230882.856483936 1436230941.208484650 0.000118494 58.351872206 0.000128508
   10 StatSess     c 54.88.194.254 52474 58 1 1 0 0 1 565 74746

I've checked that I can curl the healthcheck file after this error occurs, and it seems that I can:

[root@varnish1:/etc/varnish] #curl http://wiki.example.com/healthcheck.php
good

It's an odd setup, but it'll be a couple weeks before we'll have approval to setup varnish in in EC2. It's a setup I inherited and need to try to make it workable for the next couple of weeks.

I'd appreciate any help and insight you can provide.

Upvotes: 2

Views: 5044

Answers (2)

perbu
perbu

Reputation: 440

"errno = Connection reset by peer"

The peer is reseting the connection. This isn't happening in Varnish. Go debug the peer.

Upvotes: 4

Redithion
Redithion

Reputation: 1006

I think you have to move the set req.backend = www; statement to the first line of vcl_recv.

Now you are returning pass but no backend has been set, so I assume Varnish timeouts waiting for a backend that not exists respond.

Upvotes: 0

Related Questions