CODEWITHSUNDEEP
debuggingnginx
Soroush
Soroush

Reputation: 73

Nginx process is dead. why and where to look?

recently the nginx on my server goes down after some time about 2 times a day and does not exist or show up in processes.

I checked the nginx error file and found nothing special. just sometimes it gets "connection refused while connecting to upstream" which is ok.(it is because I took down an app and it does not answer nginx ) the access logs are ok too. nothing odd in access logs.

I checked the systemctl nginx status and it gives me this: systemctl nginx status result

which does not provide any info about why nginx is down. it just says nginx is inactive (dead) which i know it myself. also it says that ExecStop has failed. this line appears when nginx is running successfully so i dont think it is important.

i checked kern.log and no out of memory situation appeared to kill nginx. (i have more than enough memory for my apps and nginx and other stuff)

so why my nginx goes down? where else should I look for possible problems?

Upvotes: 1

Views: 4438

Answers (1)

Victor M
Victor M

Reputation: 46

Just posting to report that I am getting a similar error as well. However, mine is only occurring once per day, a couple of minutes after midnight. I am running Ubuntu 16.0.4 LTS. If I restart the service, everything works great until midnight of the next day.

Your post was the first piece of information that resembled the same patterns so I hope someone can point us in the right direction. I will report back if I find anything useful until then.

Picture: Error

-----UPDATE------

I'm pretty confident I found a resolution for my issue! However, I can't confirm until tomorrow after the clock strikes midnight :)

Confirmation of issue:

With journalctl i found that the service was STOPPING itself at midnight, then an error came up with "Service could not reload"...the keyword here is "reload". With this, I wondered why it was sending a RELOAD if the service has already been STOPPED....hmm.... So to confirm, I manually stopped my nginx service with "sudo systemctl stop nginx" then ran "sudo systemctl reload nginx" and i replicated the same error as before!

So basically, the problem is that Nginx is trying to RELOAD itself when it should be trying to RESTART itself.

Troubleshooting:

Since Nginx uses /etc/init.d/nginx as it's Upstart script when performing the systemctl commands, I found that the likely problem is that I installed my Nginx package from an outdated repo that contained an incorrect (though logical) Upstart script. To confirm this, I went over to Nginx's official examples, and opened the "Debian PHP-FPM (Lenny/Squeeze)" example (https://www.nginx.com/resources/wiki/start/topics/examples/initscripts/) and when I compared my init.d script with the example, I found two lines differed! And wouldn't you know it, the two differing lines were the reload and restart functions!

Picture: Incorrect

Picture: Correct

With this, I simply edited /etc/init.d/nginx to only have the "force-reload" case after the Restart() function, and removed "force-reload" case after the Reload() function.

TLDR: The Nginx installation contained an incorrect Upstart script which called a reload() instead of a restart(). Modify /etc/init.d/nginx to call a Restart() function when receiving a "force-reload" case instead of reload()

-----Suggestions----- OP: what I would suggest is to use journalctl -u nginx.service and see if there is a pattern to the service becoming inactive. systemctl status only shows the most recent entries, so journalctl would be a better option to look into. If you'd like, post your output here so we can look further into it.

Upvotes: 3

Related Questions