Docker Registry behind TLS enabled reverse proxy (Traefik) - Remote Error: Bad Certificate

Question

So I am doing everything Dockerized here. Traefik is running in a container, as is my docker Registry instance. I am able to push/pull just fine from the registry if I hit it at mydomain.com:5000/myimage.

The problem comes when I try to hit it through 443 using mydomain.com/myimage. The setup I have here is Traefik reverse proxy listening on 443 at mydomain.com, and forwarding that request internally to :5000 of my Registry instance.

When I go to push/pull from the Traefik url, it hangs and counts down waiting to retry on a loop. When I look at the logs of Registry, each I can see the instance IS in fact in communication with the reverse proxy Traefik, however, I get this error in the log over and over (on each push retry from the client side):

2018/05/31 21:10:43 http: TLS handshake error from proxy_container_ip:port: remote error: tls: bad certificate

Docker Registry is really tight and strict when it comes to the TLS issue. I'm using all self signed certs here, as I'm still in development. Any idea what is causing this error? I'm assuming that either the Traefik proxy detects that the certificate offered from Registry is not to be trusted (self-signed), and therefore does not complete the "push" request, or the other way around - Registry, when sending the response back through to the Traefik proxy detects that it is not to be trusted.

I can provide additional information if needed. Current setup is that both Traefik and Registry have their own set of .crt and .key files. Both (of course) TLS enabled.

Thanks.

Answer 1

Here is a working solution with a self-signed certificate that you can try out on https://labs.play-with-docker.com

Server

Add a new instance node1 in your Docker playground. We configure it as our server. Create a directory for the certificates:

mkdir /root/certs

Create wildcard certificate *.domain.local:

$ openssl req -newkey rsa:2048 -nodes -keyout /root/certs/domain.local.key -x509 -days 365 -out /root/certs/domain.local.crt
Generating a 2048 bit RSA private key
...........+++
...........+++
writing new private key to '/root/certs/domain.local.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:*.domain.local
Email Address []:

Create two files docker-compose.yml and traefik.toml in directory /root. You can download them using:

wget https://gist.github.com/maiermic/cc9c9aab939f7ea791cff3d974725e4a/raw/8c5d787998d33c752f2ab369a9393905780d551c/docker-compose.yml
wget https://gist.github.com/maiermic/cc9c9aab939f7ea791cff3d974725e4a/raw/8c5d787998d33c752f2ab369a9393905780d551c/traefik.toml

docker-compose.yml

version: '3'

services:
  frontproxy:
    image: traefik
    command: --api --docker --docker.swarmmode
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./certs:/etc/ssl:ro
      - ./traefik.toml:/etc/traefik/traefik.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
    deploy:
      labels:
        - traefik.port=8080
        - traefik.frontend.rule=Host:traefik.domain.local
  docker-registry:
    image: registry:2
    deploy:
      labels:
        - traefik.port=5000 # default port exposed by the registry
        - traefik.frontend.rule=Host:registry.domain.local
        - traefik.frontend.auth.basic=user:$$apr1$$9Cv/OMGj$$ZomWQzuQbL.3TRCS81A1g/ # user:password, see https://docs.traefik.io/configuration/backends/docker/#on-containers

traefik.toml

defaultEntryPoints = ["http", "https"]

# Redirect HTTP to HTTPS and use certificate, see https://docs.traefik.io/configuration/entrypoints/
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/ssl/domain.local.crt"
      keyFile = "/etc/ssl/domain.local.key"

# Docker Swarm Mode Provider, see https://docs.traefik.io/configuration/backends/docker/#docker-swarm-mode
[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "docker.localhost"
watch = true
swarmMode = true

Initialize Docker Swarm (replace <ip-of-node1> with the IP address of node1, for example 192.168.0.13):

docker swarm init --advertise-addr <ip-of-node1>

Deploy traefik and Docker registry:

docker stack deploy myregistry -c ~/docker-compose.yml

Client

Since we don't have a DNS server, we change /etc/hosts (replace <ip-of-node1> with the IP address of our server node1, for example 192.168.0.13):

echo "<ip-of-node1>   registry.domain.local traefik.domain.local" >> /etc/hosts

You should be able now to request the health status from traefik

$ curl -ksS https://traefik.domain.local/health | jq .
{
  "pid": 1,
  "uptime": "1m37.501499911s",
  "uptime_sec": 97.501499911,
  "time": "2018-07-19 07:30:35.137546789 +0000 UTC m=+97.600568916",
  "unixtime": 1531985435,
  "status_code_count": {},
  "total_status_code_count": {},
  "count": 0,
  "total_count": 0,
  "total_response_time": "0s",
  "total_response_time_sec": 0,
  "average_response_time": "0s",
  "average_response_time_sec": 0
}

and you should be able to request all images (none) from our registry

$ curl -ksS -u user:password https://registry.domain.local/v2/_catalog | jq .
{
  "repositories": []
}

Let's configure docker on our client. Create the directory for the registry certificates:

mkdir -p /etc/docker/certs.d/registry.domain.local/

Get the certificate from our server:

scp root@registry.domain.local:/root/certs/domain.local.crt /etc/docker/certs.d/registry.domain.local/ca.crt # Are you sure you want to continue connecting (yes/no)? yes

Now you should be able to login to our registry and add an image:

docker login -u user -p password https://registry.domain.local
docker pull hello-world:latest
docker tag hello-world:latest registry.domain.local/hello-world:latest
docker push registry.domain.local/hello-world:latest

If you request all images from our registry after that, you should see

$ curl -ksS -u user:password https://registry.domain.local/v2/_catalog | jq .
{
  "repositories": [
    "hello-world"
  ]
}