CODEWITHSUNDEEP
nginxloggingdockerlogrotate
Mark Sawers
Mark Sawers

Reputation: 302

containerized nginx log rotation with logrotate

Nginx doesn't have native log rotation, so an external tool, such as logrotate, is required. Nginx presents a challenge in that the logs have to be reopened post rotation. You can send a USR1 signal to it if the pid is available in /var/run.

But when running in a docker container, the pid file is missing in /var/run (and the pid actually belongs to the host, since it is technically a host process).

If you don't reopen the logs, nginx doesn't log anything at all, though it continues to function otherwise as web server, reverse proxy, etc.

Upvotes: 7

Views: 9031

Answers (2)

metkat
metkat

Reputation: 473

If you want to run logrotate in a dedicated container (e.g to rotate both nginx logs and Rails' file log) rather than on the host machine, here's how I did it. The trickiest part by far was as above, getting the reload signals to nginx, Rails, etc so that they would create and log to fresh logfiles post-rotation.

Summary:

  • put all the logs on a single shared volume
  • export docker socket to the logrotate container
  • build a logrotate image with logrotate, cron, curl, and jq
  • build logrotate.conf with postrotate calls using docker exec API as detailed below
  • schedule logrotate using cron in the container

The hard part:

To get nginx (/etcetera) to reload thus connect to fresh log files, I sent exec commands to the other containers using Docker's API via socket. It expects a POST with the command in JSON format, to which it responds with an exec instance ID. You then need to explicitly run that instance.

An example postrotate section from my logrotate.conf file:

postrotate
    exec_id=`curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"cmd": ["nginx", "-s", "reopen"]}' \
      http:/v1.41/containers/hofg_nginx_1/exec  \
    | jq -r '.Id'`
    curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"Detach": true}' \
      http:/v1.41/exec/"$exec_id"/start
endscript

Commentary on the hard part:

  exec_id=`curl -X POST --unix-socket /var/run/docker.sock \

This is the first of two calls to curl, saving the result into a variable to use in the second. Also don't forget to (insecurely) mount the socket into the container, '/var/run/docker.sock:/var/run/docker.sock'

  -H "Content-Type: application/json" \
  -d '{"cmd": ["nginx", "-s", "reopen"]}' \

Docker's API docs say the command can be a string or array of strings, but it only worked for me as an array of strings. I used the nginx command line tool, but something like 'kill -SIGUSR1 $(cat /var/run/nginx.pid)' would probably work too.

  http:/v1.41/containers/hofg_nginx_1/exec  \

I hard-coded the container name, if you're dealing with something more complicated you're probably also using a fancier logging service

  | jq -r '.Id'`

The response is JSON-formatted, I used jq to extract the id (excuse me, 'Id') to use next.

  curl -X POST --unix-socket /var/run/docker.sock \
      -H "Content-Type: application/json" \
      -d '{"Detach": true}' \

The Detach: true is probably not necessary, just a placeholder for POST data that was handy while debugging

  http:/v1.41/exec/"$exec_id"/start

Making use of the exec instance ID returned by the first curl to actually run the command.

I'm sure it will evolve (say with error handling), but this should be a good starting point.

Upvotes: 0

Mark Sawers
Mark Sawers

Reputation: 302

You can get the process id from the Pid attribute using docker inspect and use kill -USR1 {pid} to have nginx reopen the logs.

Here's the /etc/logrotate.d/nginx file I created:

/var/log/nginx/access.log
{
    size 2M
    rotate 10
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        docker inspect -f '{{ .State.Pid }}' nginx | xargs kill -USR1
    endscript
}

Upvotes: 8

Related Questions