CODEWITHSUNDEEP
dockernginxdocker-compose
John Howard
John Howard

Reputation: 64135

Nginx Reverse Proxy with Dynamic Containers

I have a reverse proxy with nginx set up using docker compose. It is fully working when I run all services together with docker-compose up. However, I want to be able to run individual containers, and start (docker-compose up service1) and stop them independently from the proxy container. Here is a snippet from my current nginx config:

server {
  listen 80;

  location /service1/ {
    proxy_pass http://service1/;
  }

  location /service2/ {
    proxy_pass http://service2/;
  }
}

Right now if I run service1, service2, and the proxy together all is well. However, if I run the proxy and only service2, for example, I get the following error: host not found in upstream "service1" in /etc/nginx/conf.d/default.conf:13. The behavior I want here is to just throw some HTTP error, and when that service does come up to route to it appropriately.

Is there any way to get this behavior?

Upvotes: 1

Views: 1836

Answers (3)

zok
zok

Reputation: 176

Since NGINX v1.27.3 (released 2024-11-26) to achieve proper routing when service become available you can enable dynamic hostname resolving by defining upstream servers with resolve keyword. Example:

# 1) by default NGINX use IP from /etc/resolv.conf implicitly,
# in order to enable dynamic resolving of upstream services, we need to set DNS explicitly
# 2) additionally we can set lower validity time for DNS cache to recover faster
resolver 127.0.0.11 valid=3s;

upstream upstream_service1 {
    # server group must reside in the shared memory to use "resolve":
    zone service1_zone 64k;
    # "resolve" enables IP changes monitoring and 
    #  allows NGINX to start even if upstream is not available
    server service1:80 resolve;
}

upstream upstream_service2 {
    zone service2_zone 64k;
    server service2:80 resolve;
}

server {
    # ...some other directives you have
    location /service1/ {
        proxy_pass http://upstream_service1/;
    }
    location /service2/ {
        proxy_pass http://upstream_service2/;
    }
}

With such config not only NGINX can start with unavailable upstream, but also NGINX will recover if upstream received new IP after container's restart (it happens at least if any IP was taken or released during container up time).

An old fashion alternative may looks like this:

resolver 127.0.0.11 valid=3s;

server {
    location ~ ^/service1/(.*)$ {
        # NGINX cannot resolve upstream hostname when starting
        # if we pass it through variable,
        # so NGINX will start even if upstream server is not available.
        #
        # NOTE: 
        # when we use variable, part or URL path which
        # matched location won't be truncated and would be passed to 
        # upstream as is, starting with `/servive1/...`,  so we have 
        # - to switch location to regex 
        # - and pass the rest or URL explicitly with $1$is_args$args
        set $upstream_service "http://service1:80";
        proxy_pass $upstream_service/$1$is_args$args;
    }
}

But my experiments with this older approach showed that if service IP changes, it won't be updated and we'll have 502 for this service until we send reconfigure signal to NGINX or restart it even though we have resolver directive with valid=3s.

Therefore, it seems like in that case it is not a dynamic resolving, but postponed and once hostname resolved through DNS it would be cached.

I see that it is quite old topic, but today I have spent 4 hours on this problem while most of the answers was out of date. I hope my answer may save time for someone else.

Upvotes: 0

moebius
moebius

Reputation: 2269

Your issue is with nginx. It will fail to start if it cannot resolve one of the upstream hostnames.

In your case the docker service name will be unresolvable if the service is not up.

Try one of the solutions here, such as resolving at the location level.

(edit) The below example works for me:

events {
  worker_connections  4096;
}

http {
  server {
    location /service1 {
        resolver 127.0.0.11;
        set $upstream http://service1:80;
        proxy_pass    $upstream;
    }

    location /service2 {
        resolver 127.0.0.11;
        set $upstream2 http://service2:80;
        proxy_pass    $upstream2;
    }
  }
}

Upvotes: 2

danday74
danday74

Reputation: 56966

Sounds like you need to use load balancing. I believe with load balancing it will attempt to share the load across servers/services. If one goes down, it should automatically use the others.

Example

http {
    upstream myapp1 {
        server srv1.example.com;
        server srv2.example.com;
        server srv3.example.com;
    }

    server {
        listen 80;
        location / {
            proxy_pass http://myapp1;
        }
    }
}

Docs: http://nginx.org/en/docs/http/load_balancing.html

Upvotes: 0

Related Questions