CODEWITHSUNDEEP
loggingnginxhealth-monitoring
honorableruler
honorableruler

Reputation: 79

How can I see unavailable servers in Nginx logs?

Where in the Nginx logs will it say that a server is unavailable because it failed x times in y seconds?

I have a set of servers in an upstream block in nginx, each one has a fail_timeout and max_fails value set like so:

upstream loadbalancer {
    server ip1:80 max_fails=3 fail_timeout=60s;
    server ip2:80 max_fails=3 fail_timeout=60s;
}

If I intentionally bring down one of these servers (let's say ip:80), NGINX gets back a 503 which I have marked as an invalid header. So I make sure NGINX hits that server three times in sixty seconds.

I expect there to be something in the logs that the server is being marked as unavailable, i.e. that the fail_timeout has kicked in. But I can't find anything.

Here is my logging config:

access_log /var/log/nginx/access.log  main; 
error_log /var/log/nginx/error.log warn;

Upvotes: 4

Views: 3421

Answers (3)

pupkinsen
pupkinsen

Reputation: 251

There is now a log message when the server has exceeded max_fails. It has been added in 1.9.1. Log level is warning, the message says "upstream server temporarily disabled".

Upvotes: 2

Miguel
Miguel

Reputation: 68

I am not sure if you can get logs of unavailable servers. But you can for example execute the lsof command to get additional list log files of your httpd root by its PID.

1) First xecute this command to get the PID of you HTTPD root: 

  > ps axu |grep httpd

2) Then Copy the PID of root. Lets say the PID is 1234.

3) Next we use the PID 1234 and execute the final command in order to to get the log files of the httpd root:

  > lsof -p 1234 |grep log

This has really helped me a lot finding missing logs. Now you can then check if the log files contains anything regarding unavailable servers. Best of Luck

Upvotes: 1

Steve E.
Steve E.

Reputation: 9353

You should see logging in the error log with useful information on the reason. Here are some examples from Nginx 1.8 

 [error] 9369#0: *837 connect() failed (111: Connection refused) while connecting to upstream

 [error] 9369#0: *851 connect() failed (113: No route to host) while connecting to upstream

 [error] 9369#0: *844 no live upstreams while connecting to upstream

As you can see, the log level is error so that is not an issue in your config.

You mention setting a 503 header to mark hosts as unavailable. This will not be detected in the default Nginx settings. To use specific response codes to determine upstream host status look into the proxy_next_upstream option.

Setting it to the following would include 503 response codes in the list of results that counts as an upstream failure:

proxy_next_upstream error timeout http_503;

From the documentation:
The directive also defines what is considered an unsuccessful attempt of communication with a server. The cases of error, timeout and invalid_header are always considered unsuccessful attempts, even if they are not specified in the directive. The cases of http_500, http_502, http_503 and http_504 are considered unsuccessful attempts only if they are specified in the directive. The cases of http_403 and http_404 are never considered unsuccessful attempts

Upvotes: 1

Related Questions