Why Traefik discovers container IP address on backend network?

Question

Using Traefik basic example with 2 services: traefik itself and traefik/whoami, I assigned both services to a network called web and the latter to backend network as well.

However, Traefik discovers backend IP address for traefik/whoami service and is unable to make contact via web network and results in "Gateway Timeout"

In attempt to force Traefik to use web network, i added --providers.docker.network=web and traefik.docker.network=web as well, but still it discovers backend IP address.

How can I make Traefik to discover the service IP address on the correct network?

Here's the docker-compose.yml file I'm trying to get to work.

version: "3.8"

services:

  traefik:
    image: "traefik:v2.3"
    command:
      - --log.level=DEBUG
      - --api.insecure=true
      - --api.dashboard=true
      - --api.debug=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --providers.docker.network=web
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - web

  whoami:
    image: "traefik/whoami"
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami.rule=Host(`whoami.localhost`)
      - traefik.http.routers.whoami.entrypoints=web
      - traefik.docker.network=web
    networks:
      - web
      - backend

networks:
  web:
  backend:

PS! When you remove the backend network from the service, it works fine!

Answer 1

There is one little glitch with the network names. As you are running in a compose file and don't reference an external network, the network names are created in the scope of the current compose project. When starting up, you might see some message saying

Creating network "traefik2-test_web" with the default driver
Creating network "traefik2-test_backend" with the default driver

(assuming the folder you are running in is called traefik2-test) This means the docker networks created are prefixed with the project name, which e.g. allows to spin up the compose file twice for isolated instances.

You might also see some error in the logs indicating that the configured network named web was not found - hence it falls back to randomly choosing one.

traefik_1  | time="2020-10-29T14:24:25Z" level=warning msg="Could not find network named 'web' for container '/traefik2-test_whoami_1'! Maybe you're missing the project's prefix in the label? Defaulting to first available network." providerName=docker container=whoami-traefik2-test-4661d09157cf941b7efd7c8154af5611249866c4bc394c256e225e05db21a784 serviceName=whoami-traefik2-test

So traefik expects the globally valid name for the network and doesn't care about that it was started from inside the compose project. You can validate the network names by doing a

docker container inspect XXX_whoami_1

where XXX is the name of your docker-compose project.

What can you do now?

Solution 1: fix and reference the project name via env

Construct the network name accordingly, so that the param matches the globally unique name for the network that docker-compose creates. If all your target containers in this compose file are in the web network, you can even omit the special label on the target containers.

    traefik:
      image: "traefik:v2.3"
      command:
        - --log.level=DEBUG
        - --api.insecure=true
        - --api.dashboard=true
        - --api.debug=true
        - --providers.docker=true
        - --providers.docker.exposedbydefault=false
        - --entrypoints.web.address=:80
        - --providers.docker.network=${COMPOSE_PROJECT_NAME}_web
      ports:
        - "80:80"
        - "8080:8080"
      volumes:
        - "/var/run/docker.sock:/var/run/docker.sock:ro"
      networks:
        - web

This requires you to pass a docker compose project name for each and every command (or have the env set accordingly)

COMPOSE_PROJECT_NAME=proj docker-compose up

The issue with that is that you really need to specify that variable for each and every call. You might be able to fix that by using an .env file next to the docker-compose.yml that enables that.

COMPOSE_PROJECT_NAME=projfromenv

Alternative solution: external network reference

But there might be a better solution. That second option is to use a globally unique network name by referencing an existing network. This needs initial setup of the network like so:

docker network create traefik-net

And then referencing it in the networks section:

networks:
  web:
    external: true
    name: traefik-net
  backend:

And starting traefik with the reference to this network name (snippet):

- --providers.docker.network=traefik-net

Your target containers still reference the local network name web which is now a reference to that global one. You should not need to specifiy the traefik network label for the whoami containers because you don't need to overwrite the default.

This setup also allows to spin up other compose projects with containers that also should get exposed from this traefik instance. So I'd go with that second option...