CODEWITHSUNDEEP
dockernginxdocker-swarmnginx-reverse-proxy
sas
sas

Reputation: 7542

nginx does not automatically pick up dns changes in swarm

I'm running nginx via lets-nginx in the default nginx configuration (as per the lets-nginx project) in a docker swarm:

services:

  ssl:
    image: smashwilson/lets-nginx
    networks:
      - backend
    environment:
      - [email protected]
      - DOMAIN=api.finestructure.co
      - UPSTREAM=api:5000
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - letsencrypt:/etc/letsencrypt
      - dhparam_cache:/cache

  api:
    image: registry.gitlab.com/project_name/image_name:0.1
    networks:
      - backend
    environment:
      - APP_SETTINGS=/api.cfg
    configs:
      - source: api_config
        target: /api.cfg
    command:
      - run
      - -w
      - tornado
      - -p
      - "5000"

api is a flask app that runs on port 5000 on the swarm overlay network backend.

When services are initially started up everything works fine. However, whenever I update the api in a way that makes the api container move between nodes in the three node swarm, nginx fails to route traffic to the new container.

I can see in the nginx logs that it sticks to the old internal ip, for instance 10.0.0.2, when the new container is now on 10.0.0.4.

In order to make nginx 'see' the new IP I need to either restart the nginx container or docker exec into it and kill -HUP the nginx process.

Is there a better and automatic way to make the nginx container refresh its name resolution?

Upvotes: 8

Views: 4901

Answers (2)

Ser
Ser

Reputation: 2870

Remember that when you use set you need to generate the entire URL by yourself.

I was using nginx in a compose to proxy a zuul gateway : 

        location /api/v1/ {
           proxy_set_header X-Forwarded-Host $host:$server_port;
           proxy_pass http://rs-gateway:9030/api/v1/;
        }

        location /zuul/api/v1/ {
           proxy_set_header X-Forwarded-Host $host:$server_port;
           proxy_pass http://rs-gateway:9030/zuul/api/v1/;
        }

Now with Swarm it looks like that :

        location ~ ^(/zuul)?/api/v1/(.*)$ {
            set $upstream http://rs-gateway:9030$1/api/v1/$2$is_args$args;
            proxy_pass $upstream;
            # Set headers
            proxy_set_header X-Forwarded-Host $host:$server_port;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $http_connection;
        }

Regex are good but don't forget to insert GET params into the generated URL by yourself.

Upvotes: 0

sas
sas

Reputation: 7542

Thanks to @Moema's pointer I've come up with a solution to this. The default configuration of lets-nginx needs to be tweaked as follows to make nginx pick up IP changes:

  resolver 127.0.0.11 ipv6=off valid=10s;
  set $upstream http://${UPSTREAM};
  proxy_pass $upstream;

This uses docker swarm's resolver with a TTL and sets a variable, forcing nginx to refresh name lookups in the swarm.

Upvotes: 13

Related Questions