Docker for Windows - Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden

Question

I try to run Kubernetes from Docker for Windows. After I click on Enable Kubernetes inside the Kubernetes Tab the Kuberneters is starting... process running into a endless state.

Take look in the service.txt log in C:\ProgramData\DockerDesktop\pki, Docker repeat the following log-block for the whole time.

[10:23:26.068][ApiProxy          ][Error  ] time="2020-01-14T10:23:26+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:27.080][ApiProxy          ][Error  ] time="2020-01-14T10:23:27+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:28.071][ApiProxy          ][Error  ] time="2020-01-14T10:23:28+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:28.624][ApiProxy          ][Info   ] time="2020-01-14T10:23:28+01:00" msg="DNS failure: www-cache.\tIN\t A: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:28.626][ApiProxy          ][Info   ] time="2020-01-14T10:23:28+01:00" msg="DNS failure: www-cache.\tIN\t AAAA: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:29.068][ApiProxy          ][Error  ] time="2020-01-14T10:23:29+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:30.083][ApiProxy          ][Error  ] time="2020-01-14T10:23:30+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:31.088][ApiProxy          ][Error  ] time="2020-01-14T10:23:31+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:32.068][ApiProxy          ][Error  ] time="2020-01-14T10:23:32+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:32.715][ApiProxy          ][Info   ] time="2020-01-14T10:23:32+01:00" msg="DNS failure: www-cache.\tIN\t AAAA: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:32.717][ApiProxy          ][Info   ] time="2020-01-14T10:23:32+01:00" msg="DNS failure: www-cache.\tIN\t A: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:33.068][ApiProxy          ][Error  ] time="2020-01-14T10:23:33+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:34.074][ApiProxy          ][Error  ] time="2020-01-14T10:23:34+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:34.658][ApiProxy          ][Info   ] time="2020-01-14T10:23:34+01:00" msg="DNS failure: www-cache.\tIN\t A: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:34.661][ApiProxy          ][Info   ] time="2020-01-14T10:23:34+01:00" msg="DNS failure: www-cache.\tIN\t AAAA: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:35.069][ApiProxy          ][Error  ] time="2020-01-14T10:23:35+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:36.074][ApiProxy          ][Error  ] time="2020-01-14T10:23:36+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:37.070][ApiProxy          ][Error  ] time="2020-01-14T10:23:37+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:38.072][ApiProxy          ][Error  ] time="2020-01-14T10:23:38+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:39.072][ApiProxy          ][Error  ] time="2020-01-14T10:23:39+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:39.681][ApiProxy          ][Info   ] time="2020-01-14T10:23:39+01:00" msg="DNS failure: www-cache.\tIN\t AAAA: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:39.684][ApiProxy          ][Info   ] time="2020-01-14T10:23:39+01:00" msg="DNS failure: www-cache.\tIN\t A: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:40.069][ApiProxy          ][Error  ] time="2020-01-14T10:23:40+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:41.076][ApiProxy          ][Error  ] time="2020-01-14T10:23:41+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:42.089][ApiProxy          ][Error  ] time="2020-01-14T10:23:42+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:42.745][ApiProxy          ][Info   ] time="2020-01-14T10:23:42+01:00" msg="DNS failure: www-cache.\tIN\t A: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:42.748][ApiProxy          ][Info   ] time="2020-01-14T10:23:42+01:00" msg="DNS failure: www-cache.\tIN\t AAAA: errno 9002: DnsQuery: DNS-Serverfehler."
[10:23:43.071][ApiProxy          ][Error  ] time="2020-01-14T10:23:43+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:44.088][ApiProxy          ][Error  ] time="2020-01-14T10:23:44+01:00" msg="Cannot list nodes: Get https://kubernetes.docker.internal:6443/api/v1/nodes: Forbidden"
[10:23:44.758][VpnKit            ][Info   ] vpnkit.exe: Expired 256 UDP NAT rules

---

Trouble Shooting:

Proxy-Settings

my machine is behind a proxy so I add the corresponding informations inside the Proxy tab

No changes

Ping kubernetes.docker.internal

Ping wird ausgeführt für kubernetes.docker.internal [127.0.0.1] mit 32 Bytes Daten:
Antwort von 127.0.0.1: Bytes=32 Zeit<1ms TTL=128
Antwort von 127.0.0.1: Bytes=32 Zeit<1ms TTL=128
Antwort von 127.0.0.1: Bytes=32 Zeit<1ms TTL=128
Antwort von 127.0.0.1: Bytes=32 Zeit<1ms TTL=128
Ping-Statistik für 127.0.0.1:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0

Proxy: Ignore Local Address

Therefore kubernetes.docker.internal is a local address I add the address to the proxy ignoring list in docker and in my machine internetoptions.

No changes

Install ca.crt from C:\ProgramData\DockerDesktop\pki

I also try to add the docker .crt to the trusted certificates of my machine

No changes

Remove PKI and Reset Kubernetes Cluster

the endless state of starting kubernetes is not rare, so I found a lot suggestions to handle on github. The most working suggestions are about remove stuff and reseting docker. I try all of them multiple times.

No changes

Call https://kubernetes.docker.internal:6443/api/v1/nodes in Browser

---

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:37 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Testing on a Windows 10 Machine.

---

I spread all my shots and no more clue what to do.

Answer 1

4 years after the original question was posted ... let me share what I came across recently with "failing to start docker kubernetes".

For whatever reason, I would only be able to start a kubernetes cluster from docker desktop after a fresh install ... after using it once and restarting my computer, it was enough to get the kubernetes cluster on a never ending loop of "starting" and "failing to start".

Long story short, I've noticed my WSL integration wasn't running.

You can get a list of available wsl environments with the following command:

wsl -l -v

mine returns something like this

NAME                  STATE     VERSION
Ubuntu                Running   2
docker-desktop-data   Stopped   2
docker-desktop        Stopped   2

At this point I went for a reboot to get things cleared out and fresh. Once I was back in W11 ... this was when I decided to get those wsl integrations running manually

wsl -d docker-desktop-data
wsl -d docker-desktop

And THEN, and only then, get Docker desktop started. Kubernetes started without a hitch.

Hope this helps!

Answer 2

I'm having the same problem and it seems, that the k8s API doesn't want to answer to the TLS Client hello message. I checked the traffic with Wireshark on the local interface (this is the one used for kubernetes.docker.internal). The TCP session setup is working properly.

I also checked in the "Show system containers (advanced)" option in the docker for windows settings under the kubernetes tab, but the "docker ps -a" does not show up any container (I'm not sure it should, but the option's name suggest that to me).

I would gladly continue the debugging and see whether the API service is actually running in the HyperV virtual machine that provides docker in Windows, but I'm not able to connect to it through the Hyper-V Manager. Any idea, how to check that and get the logs for the service?

Answer 3

I highly recommend you to get K8s up when Windows Firewall fully OFF AND connected to a home network. Booting Docker & K8s while connected to the corporate network causes it to hang again at "Kubernetes is starting..."

Another solution

1. Change DNS to fixed and use 8.8.8.8, this is within docker for window's settings

2. Remove the .kube

3. Add the KUBECONFIG environment variable to System Variables and have the path be C:\Users[MYUSER].kube\config. Note that before I had it set as a User Variable.

4. Restart Docker from the Docker for Window's reset tab in settings.

5. Restart Kubernetes Cluster from the Docker for Window's reset tab in settings (you can do this a number of times).

Afterwards just wait for some time and Kubernetes is running should display

Take a look here: kubernetes-fails-to-start.

I hope it helps.