CODEWITHSUNDEEP
dockernginxdns
blueFast
blueFast

Reputation: 44381

Inconsistent name resolution with nginx and docker compose

I have been testing this for a couple of hours now. I am pretty certain that nginx is not resolving the host names consistently when running in a docker environment. Specifically, I have this kind of config:

location /api/ {
    proxy_pass         http://api:8000;

    proxy_set_header   Host                 $http_host;
    proxy_set_header   X-Real-IP            $remote_addr;
    proxy_set_header   X-Forwarded-For      $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto    $scheme;
}

I see requests going sometimes to the correct service (the api service), and sometimes it ends up in a completely different service. This happens after restarting the services?!?

I am using the default network (no custom network), with docker-compose for managing the services.

Is this a known problem? Can I do something to improve the situation?

This is my nginx:

» docker exec -it nginx_1 bash
root@3ebfda9b7c98:/# nginx -V       
nginx version: nginx/1.19.4
built by gcc 8.3.0 (Debian 8.3.0-6) 
built with OpenSSL 1.1.1d  10 Sep 2019
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.19.4/debian/debuild-base/nginx-1.19.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
root@3ebfda9b7c98:/#

And this is my docker:

» docker --version
Docker version 19.03.13, build 4484c46d9d

Update

I think the culprit is somehow the nginx application itself: if I go inside the nginx container and resolve manually the service (ping api), it always works as expected.

It seems as if somehow nginx does sometimes a bad resolution and sticks to it.

Upvotes: 2

Views: 597

Answers (1)

blueFast
blueFast

Reputation: 44381

After some back and forth, this is how I solved this problem:

In order to make nginx aware of upstream changes (properly resolve hostnames when containers move around), we abuse an nginx feature: hostnames used literally are resolved on startup, but hostnames used via variables are resolved using the nginx internal resolver, which includes a ttl parameter (valid).

This is the described in the documentation for the proxy_pass directive.

This is the documentation for the resolver directive

For example:

server {

    listen 80;
    server_name localhost;

    ...

    # The docker resolver, needed when using variables for the hostnames
    resolver 127.0.0.11 ipv6=off valid=10s;

    location /api/ {
        # Defining a variable here will force nginx to do periodic name resolution
        set                $gateway             api:1234;
        proxy_pass         http://$gateway;

        proxy_set_header   Host                 $http_host;
        proxy_set_header   X-Real-IP            $remote_addr;
        proxy_set_header   X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $scheme;
    }

    ...

}

Upvotes: 1

Related Questions