kubeadm init shows kubelet isn't running or healthy

Question

I am trying to run Kubernetes and trying to use sudo kubeadm init. Swap is off as recommended by official doc.

The issue is it displays the warning:

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.


Unfortunately, an error has occurred:
            timed out waiting for the condition

This error is likely caused by:
            - The kubelet is not running
            - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
            - No internet connection is available so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-amd64:v1.11.2
                - k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
                - k8s.gcr.io/kube-scheduler-amd64:v1.11.2
                - k8s.gcr.io/etcd-amd64:3.2.18
                - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
                  are downloaded locally and cached.

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
            - 'systemctl status kubelet'
            - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
        Here is one example how you may list all Kubernetes containers running in docker:
            - 'docker ps -a | grep kube | grep -v pause'
            Once you have found the failing container, you can inspect its logs with:
            - 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

The docker version I am using is Docker version 17.03.2-ce, build f5ec1e2 I m using Ubuntu 16.04 LTS 64bit

The docker images shows the following images:

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver-amd64            v1.11.2             821507941e9c        3 weeks ago         187 MB
k8s.gcr.io/kube-controller-manager-amd64   v1.11.2             38521457c799        3 weeks ago         155 MB
k8s.gcr.io/kube-proxy-amd64                v1.11.2             46a3cd725628        3 weeks ago         97.8 MB
k8s.gcr.io/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        3 weeks ago         56.8 MB
k8s.gcr.io/coredns                         1.1.3               b3b94275d97c        3 months ago        45.6 MB
k8s.gcr.io/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219 MB
k8s.gcr.io/pause                           3.1                 da86e6ba6ca1        8 months ago        742 kB

Full logs can be found here : https://pastebin.com/T5V0taE3

I didn't found any solution on internet.

EDIT:

docker ps -a output:

ubuntu@ubuntu-HP-Pavilion-15-Notebook-PC:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS

journalctl -xeu kubelet output:

journalctl -xeu kubelet
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: T
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: M
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: U
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: F
lines 788-810/810 (END)
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117131    9107 server.go:408] Version: v1.11.2
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117406    9107 plugins.go:97] No cloud provider specified.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.121192    9107 certificate_store.go:131] Loading cert/key pair 
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.145720    9107 server.go:648] --cgroups-per-qos enabled, but --
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.146074    9107 server.go:262] failed to run Kubelet: Running wi
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Unit entered failed state.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Failed with result 'exit-code'.
~


          PORTS               NAMES

Any help/suggestion/comment would be appreciated.

Answer 1

I faced similar issue recently. The problem was cgroup driver. Kubernetes cgroup driver was set to systems but docker was set to systemd. So I created /etc/docker/daemon.json:

vim /etc/docker/daemon.json

and added below:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

Then

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet

Run kubeadm init or kubeadm join again.

Answer 2

Unfortunately, swap was still enabled.
The error was fixed by

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

reboot the machine.

Answer 3

You can try following:

sudo apt-get install -yq kubelet=1.25.5-00 kubeadm=1.25.5-00

Answer 4

use

export KUBECONFIG=/etc/kubernetes/admin.conf

or you can create file name

nano  /etc/docker/daemon.json

and write

{
      "exec-opts": ["native.cgroupdriver=cgroupfs"],
      "log-driver": "json-file",
      "log-opts": {
      "max-size": "100m"
   },
       "storage-driver": "overlay2"
       }

then

systemctl daemon-reload
systemctl restart kubelet
systemctl status  kubelet

Answer 5

I had to modify ExecStart options in /usr/lib/systemd/system/docker.service as indicated in one of the responses here: docker change cgroup driver to systemd

And very important: I had to delete the file /etc/docker/daemon.json that I had created trying to solve this problem. Otherwise, after modifying /usr/lib/systemd/system/docker.service you get an error because the same option is also in daemon.json

Answer 6

As you use Kubernetes 1.11.2, it's useful to quote from CHANGELOG-1.11.md:

kubeadm now detects the Docker cgroup driver and starts the kubelet with the matching driver. This eliminates a common error experienced by new users in when the Docker cgroup driver is not the same as the one set for the kubelet due to different Linux distributions setting different cgroup drivers for Docker, making it hard to start the kubelet properly.

It seems to me that on your node some wrong parameter is passed to kubelet and this one doesn't start.

1. First of all check that your docker has cfgroups with the command:

   docker info | grep -i cgroup
   

Output should be:

Cgroup Driver: cgroupfs

2. Now go on your node and look for the kubelet service script, probably in /etc/systemd/system/kubelet.service (or similar name) and remove all parameters regarding cfgroup inside that script.

3. Try to restart the kubelet service

4. Check the kubelet's log again (journalctl -xeu kubelet)

Answer 7

I had this same issue when trying to initialize my k8s cluster. In my case, the error was rooting from Docker and Kubelet having inconsistent cgroups.

To solve it, first find Docker cgroup:

docker info | grep Cgroup

The result of the above command would be something like this:

Cgroup Driver: cgroupfs
Cgroup Version: 1

Then, update kubelet args (KUBELET_KUBECONFIG_ARGS) in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add a --cgroup-driver flag corresponsing to docker cgroup (in this case cgroupfs).

My config file looks like this after the modification:

...
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/e`tc/kubernetes/kubelet.conf --cgroup-driver=cgroupfs"
...

Finally, run kubeadm reset and then kubeadm init.

Answer 8

I met the exactly same problem.Turn swap off can solve this problem. But I avoided it in a different way, so I post my solution here for reference.

Swap check in kubelet have two phases. One is kubeadm command line tool,the other one is kubelet service.

So if we don't turn swap off,kubeadm should show the message below

[ERROR Swap]: running with swap on is not supported. Please disable swap

then it will abort the init or join process. By adding the parameter "--ignore-preflight-errors=Swap" to kubeadm can avoid kubeadm check. Here is an example:

sudo kubeadm join 10.50.10.198:6443 --token XXXX.XXXXXa     --discovery-token-ca-cert-hash sha256:XX48cb7c381 --ignore-preflight-errors=Swap

However, if we run this. Kubelet service will block us ,and it will occur the problem of this thread. We can add a configuration file to avoid this:

 cd /etc/systemd/system/kubelet.service.d
 touch 20-allow-swap.conf

Add these contents into this file.

[Service] 
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false”

Then

 systemctl daemon-reload
 systemctl restart kubelet

At last don't forget to run kubeadm join agin,with "--ignore-preflight-errors=Swap"

Answer 9

You should not use with swap even have been closed by swapoff -a.

You should disable in setting file /etc/fstab

for me:

root@kali:~# cat /etc/fstab

UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

after editing:

root@kali:~# cat /etc/fstab

UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

#UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

then.. reboot your machine

Note: Don't use reset after rebooting before kubeadm init via:

kubeadm reset

Answer 10

if you are running on openstack or any cloud please make sure to allow the ports in security groups TCP | Inbound | 6443* | Kubernetes API server

in my case this was the error

Answer 11

sudo sed -i '/ swap / s/^/#/' /etc/fstab

I spent last few days to solve the same problem and this also worked for me. Don't know whay it doesn't have same effect only with 'sudo swapoff -a'.

[kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

And these error logs are really confusing because there is no clue there showing that this problem may be related to swap configuraiton.