Containers with docker-compose have different behavior depending on machine where they are executed

Question

Good day,

I'm trying to make the below docker-compose working on my machine but I have issues with some 404 errors when trying to access url of container A from container B.

A priori the only thing which wasn't in the source control was a modification in the host file where I had to add the following rows.

127.0.0.1 idsrv4admin.traefik.me
127.0.0.1 idsrv4adminApi.traefik.me
127.0.0.1 login.traefik.me

I got all resources from source control and on my friend's machine it's working fine. Maybe I have a different configuration on my machine but I can't find out what it is.

If I try to access "http://login.traefik.me/.well-known/openid-configuration" directly from my browser I can access it:

{"issuer":"http://login.traefik.me","authorization_endpoint":"http://login.traefik.me/connect/authorize","token_endpoint":"http://login.traefik.me/connect/token","userinfo_endpoint":"http://login.traefik.me/connect/userinfo","end_session_endpoint":"http://login.traefik.me/connect/endsession","check_session_iframe":"http://login.traefik.me/connect/checksession","revocation_endpoint":"http://login.traefik.me/connect/revocation","introspection_endpoint":"http://login.traefik.me/connect/introspect","device_authorization_endpoint":"http://login.traefik.me/connect/deviceauthorization","frontchannel_logout_supported":true,"frontchannel_logout_session_supported":true,"backchannel_logout_supported":true,"backchannel_logout_session_supported":true,"scopes_supported":["roles","openid","profile","email","address","identity_admin_api","offline_access"],"claims_supported":["role","sub","updated_at","locale","zoneinfo","birthdate","gender","website","picture","preferred_username","nickname","middle_name","given_name","family_name","name","profile","email","email_verified","address"],"grant_types_supported":["authorization_code","client_credentials","refresh_token","implicit","password","urn:ietf:params:oauth:grant-type:device_code"],"response_types_supported":["code","token","id_token","id_token token","code id_token","code token","code id_token token"],"response_modes_supported":["form_post","query","fragment"],"token_endpoint_auth_methods_supported":["client_secret_basic","client_secret_post"],"subject_types_supported":["public"],"code_challenge_methods_supported":["plain","S256"],"request_parameter_supported":true}

If I connect from the container admin and I try a curl on the same url I have a 404 with the following error message:

* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x560c38473f50)
* Connected to login.traefik.me (127.0.0.1) port 80 (#0)
> GET /.well-known/openid-configuration HTTP/1.1
> Host: login.traefik.me
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Mon, 23 Nov 2020 08:34:58 GMT
< Content-Length: 0
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SameOrigin
< Referrer-Policy: no-referrer
< Content-Security-Policy: script-src 'self' 'unsafe-inline' 'unsafe-eval';style-src 'self' 'unsafe-inline' https://fonts.googleapis.com/ https://fonts.gstatic.com/;font-src 'self' https://fonts.googleapis.com/ https://fonts.gstatic.com/
<
* Connection #0 to host login.traefik.me left intact

Below is the docker-compose:

version: "3.4"

services:
  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.traefik.address=:9090"
    ports:
      - "80:80"
      - "443:443"
      - "9090:9090"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - proxy

  admin:
    image: ${DOCKER_REGISTRY-}admin:latest
    build:
      context: .
      dockerfile: src/IdentityServer/Admin/Dockerfile
    container_name: is4-admin
    hostname: idsrv4admin.traefik.me
    expose:      
      - '80'
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.identityserver4Admin.rule=Host(`idsrv4admin.traefik.me`)"
      - "traefik.http.routers.identityserver4Admin.entrypoints=web"
    environment:
      - VIRTUAL_HOST=idsrv4admin.traefik.me
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - DOTNET_USE_POLLING_FILE_WATCHER=1
      - "AdminConfiguration__IdentityAdminRedirectUri=http://idsrv4admin.traefik.me/signin-oidc"
      - "AdminConfiguration__IdentityServerBaseUrl=http://login.traefik.me"
      - "AdminConfiguration__RequireHttpsMetadata=false"
    depends_on:
      - sts.identity
      - admin.api
    networks:
      - proxy

  admin.api:
    image: ${DOCKER_REGISTRY-}admin-api:latest
    build:
      context: .
      dockerfile: src/IdentityServer/Admin.Api/Dockerfile
    container_name: is4-admin-api
    hostname: idsrv4adminApi.traefik.me
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.identityserver4AdminApi.rule=Host(`idsrv4adminApi.traefik.me`)"
      - "traefik.http.routers.identityserver4AdminApi.entrypoints=web"
    environment:
      - VIRTUAL_HOST=idsrv4adminApi.traefik.me
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - DOTNET_USE_POLLING_FILE_WATCHER=1
      - "AdminApiConfiguration__RequireHttpsMetadata=false"
      - "AdminApiConfiguration__ApiBaseUrl=http://idsrv4adminApi.traefik.me"
      - "AdminApiConfiguration__IdentityServerBaseUrl=http://login.traefik.me"
    depends_on:
      - sts.identity
    networks:
      - proxy

  sts.identity:
    image: ${DOCKER_REGISTRY-}sts-identity:latest
    build:
      context: .
      dockerfile: src/IdentityServer/STS.Identity/Dockerfile
    container_name: is4-sts-identity
    hostname: login.traefik.me
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.identityserver4STS.rule=Host(`login.traefik.me`)"
      - "traefik.http.routers.identityserver4STS.entrypoints=web"
    environment:
      - VIRTUAL_HOST=login.traefik.me
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - DOTNET_USE_POLLING_FILE_WATCHER=1
      - "AdminConfiguration__IdentityAdminBaseUrl=http://idsrv4admin.traefik.me"
    networks:
      - proxy

networks:
  proxy:
    driver: bridge

Anybody has an idea ?

Thank in advance

Answer 1

You are using hostnames provided by the traefik.me service which (similar to xip.io) provides a wildcard DNS service by resolving domains to IPs according to the subdomain pattern.

For example, a dns query to myapp.1.2.3.4.traefik.me will resolve to the IP address encoded in the subdomain 1.2.3.4. If you don't specify an IP address in the subdomain (and not using the hexadecimal notation, too), this service automatically resolves to 127.0.0.1 which works in most cases for host-to-container setups.

Running on your host machine, using this IP address (the localhost) is pointing to your host machine. As you have traefik exposed using docker compose, this works as the request from your local machine to your local machine is mapped to the container (due to the ports mapping you defined for traefik).

This is not the case when running inside the container itself. See the log output of your curl command:

* Connected to login.traefik.me (127.0.0.1) port 80 (#0)

Running the exact same request from inside one of the containers will also resolve to the 127.0.0.1 IP, but inside a container this IP address is no longer your host machine address, but the loopback adapter of the container network interface. So basically you are running into a pretty common issue about the meaning of 127.0.0.1 when using containers as already answered several times here:

• PHP app cannot connect to docker mysql container at 127.0.0.1
• docker-compose + django + redis - Error 111 connecting to 127.0.0.1:6379. Connection refused
• Rails app using docker-compose wont connect to the mysql container

So if you need to have one single dns name that works from the host and from the container, you need to switch to something like login.192.168.1.123.traefik.me assuming 192.168.1.123 is your host IP address. But as you notice this needs adaption for each single developer and even when you change the network/wifi you are connected to.

So, I guess there is no go-to solution for this using such a wildcard dns service if you want to use the exact same domain name for public access and inter-container calls. In a production scenario, the hostname would resolve to a globally valid public IP address that also works when accessed from inside the container as it would go a full round trip through traefik, but this is not really a good option for such a development setup.

In this scenario, you would really need to specify different domains depending on if you are issuing a redirect (using the globally valid domain) or if you are using a direct call from one container to another. Those are reachable using the docker-compose internal service names, e.g. like http://sts.identity or - less preferred - the container name (http://is4-sts-identity). Can you specify that those urls are used instead of the public ones by configuring the environment variable AdminApiConfiguration__IdentityServerBaseUrl accordingly?

Answer 2

You are trying to connect to 127.0.0.1 Connected to login.traefik.me (127.0.0.1) port 80 (#0) which will send you to 127.0.0.1 inside your admin container and not on your machine

To make it work you need to map login.traefik.me to your local address (192.168.x.x)

Also, it would be better if you call that endpoint with the service name, since all the containers are in the same network you can replace this environement variable

- "AdminApiConfiguration__IdentityServerBaseUrl=http://login.traefik.me"

With

- "AdminApiConfiguration__IdentityServerBaseUrl=http://sts.identity:80"