Kubernetes node fails to join cluster

Question

I have a new install of K8s master and node both on ubuntu-18. The master is using weave for CNI and all pods are running:

$ sudo kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-29qg5                   1/1     Running   0          31m
kube-system   coredns-6d4b75cb6d-kxxc8                   1/1     Running   0          31m
kube-system   etcd-ubuntu-18-extssd                      1/1     Running   2          31m
kube-system   kube-apiserver-ubuntu-18-extssd            1/1     Running   2          31m
kube-system   kube-controller-manager-ubuntu-18-extssd   1/1     Running   2          31m
kube-system   kube-proxy-nvqjl                           1/1     Running   0          31m
kube-system   kube-scheduler-ubuntu-18-extssd            1/1     Running   2          31m
kube-system   weave-net-th4kv                            2/2     Running   0          31m

When I execute the kubeadm join command on the node I get the following error:

sudo kubeadm join 192.168.0.12:6443 --token ikk2kd.177ij0f6n211sonl --discovery-token-ca-cert-hash sha256:8717baa3c634321438065f40395751430b4fb55f43668fac69489136335721dc
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR CRI]: container runtime is not running: output: E0724 16:24:41.009234    8391 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-07-24T16:24:41-06:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

The only problem showing up in journalctl -r -u kubelet is:

kubelet.service: Main process exited, code=exited, status=1/FAILURE
...
Error: failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml

That is from several minutes before the join failed when kubelet was trying to start. I would expect that config.yaml file to be missing until the node joined a cluster.

The preflight error message says

[ERROR CRI]: container runtime is not running: output: E0724 16:32:41.120653   10509 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

What is this trying to tell me?

====Edit===== I am running CrashPlan on the worker node that is failing, but I have fs.inotify.max_user_watches=1048576 in /etc/sysctl.conf.

This node worked before both with on-prem master and with GKE with kubernetes 1.20.

Answer 1

Obtain the same error when I attempted to join another master node.

The error was corrected in accordance with the instructions of this article.

Answer 2

[ERROR CRI]: container runtime is not running: output: E0724 16:32:41.120653 10509 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"..............

rm /etc/containerd/config.toml... systemctl restart containerd...now u can run kubeadm init command

Answer 3

Check the error message. The container runtime is down

[ERROR CRI]: container runtime is not running

run the below command to check the runtime used in the cluster

kubectl get no -owide

see the last column in the picture below

Ensure the container runtime is running, In this case it is containerd