CODEWITHSUNDEEP
dockerkubernetescluster-computing
sierralimaa
sierralimaa

Reputation: 131

Kubernetes- error uploading crisocket: timed out waiting for the condition

I am trying to create a template for a Kubernetes cluster having 1 master and 2 worker nodes. I have installed all the pre-req software and have run the kubeadmn init on my master node. But when i try to run the kubeadmn join which i get as an output of the init command i am getting an error.

[discovery] Created cluster-info discovery client, requesting info
from "https://10.31.2.33:6443" [discovery] Requesting info from
"https://10.31.2.33:6443" again to validate TLS against the pinned
public key [discovery] Cluster info signature and contents are valid
and TLS certificate validates against pinned roots, will use API
Server "10.31.2.33:6443" [discovery] Successfully established
connection with API Server "10.31.2.33:6443" [kubelet] Downloading
configuration for the kubelet from the "kubelet-config-1.12" ConfigMap
in the kube-system namespace [kubelet] Writing kubelet configuration
to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet
environment file with flags to file
"/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the
kubelet service [tlsbootstrap] Waiting for the kubelet to perform the
TLS Bootstrap... [patchnode] Uploading the CRI Socket information
"/var/run/dockershim.sock" to the Node API object "<workernode2>" as
an annotation error uploading crisocket: timed out waiting for the
condition```

I have done a swapoff -a before running this on the workdernode2

I was able to run the join once but after that, as a part of a script, I ran the kubeadmn reset followed by init and join few times where this has started showing up.

Not able to figure out what or where I am doing a mistake.

My main intent is to put all the commands in the form of a shell script (on master node) so that it can be run on a cluster to create a network.

Upvotes: 13

Views: 26631

Answers (4)

abhijeet shingankar
abhijeet shingankar

Reputation: 1

some commands provided need to run on all nodes like below.

kubeadm reset
swapoff -a => all nodes.
rm -rf /var/lib/cni/
systemctl restart kubelet
iptables -F
systemctl restart containerd
systemctl daemon-reload

Then we need to run kubeadm init & follow the instructions given in the output.

Upvotes: 0

John Mulhall
John Mulhall

Reputation: 173

I would recommend caution in altering iptables. This after enormous amount of troubleshooting on a 2 node Ubuntu 20.04 kubeadm cluster was down to internal node mapping that the CNI plugin is dependant on.

With sudo hostnamectl set-hostname c23996dd402c.mylabserver.com I changed the master node back to its mapped name and then did a sudo systemctl daemon-reload. I logged out and back in for good measure to ensure the reload worked. I could see the command prompt change back to the default VM node name.

The mapping now matches for hostname to private IP for the cluster:

~$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
# Cloud Server Hostname mapping
172.31.xx.xxx   xxxxxxxx.mylabserver.com
woopsie@master:~$ cat /etc/hostname
xxxxxxxx.mylabserver.com

The worker1 node has the same problem so reverting to system default was the easier way for me. I just changed it sudo hostnamectl set-hostname masterdefaultnodename. IMHO, iptable alterations should be always done after careful consideration and as an escalated measure to fix the issue in my view.

Master done, onto worker1...

kubectl get nodes
NAME                           STATUS     ROLES           AGE   VERSION
masterdefaultnodename.mylabserver.com   Ready      control-plane   33d   v1.25.0
worker1defautnodname.mylabserver.com   NotReady   <none>          33d   v1.25.0

Upvotes: 0

Alexz
Alexz

Reputation: 741

I’ve had the same problem on Ubuntu 16.04 amd64, fixed it with these commands:

swapoff -a    # will turn off the swap 
kubeadm reset
systemctl daemon-reload
systemctl restart kubelet
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X  # will reset iptables

Also, look at that issue in kubeadm GitHub kubeadm swap on the related issue where people still report having the problem after turning swap off.

You may also try to add --fail-swap-on=false flag in /etc/default/kubelet file, it didn't help in my case though.

It seems to be fixed in the latest k8 version because after upgrading the cluster, I haven't experienced the issue.

Upvotes: 10

Deb
Deb

Reputation: 663

I had the encountered the following issue after node was rebooted:

[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8smaster" as an annotation
[kubelet-check] Initial timeout of 40s passed.
error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition

Steps to get rid of this issue:

  1. Check the hostname again, after reboot it might have changed.

    sudo vi /etc/hostname 
sudo vi /etc/hosts

  2. Perform the following clean-up actions

    Code:

    sudo kubeadm reset
rm -rf /var/lib/cni/
sudo rm -rf /var/lib/cni/

systemctl daemon-reload

systemctl restart kubelet

sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

  3. Execute the init action with the special tag as below

    Code:

    sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=10.10.10.2 --ignore-preflight-errors=all

    (where 10.10.10.2 is the IP of master node and 192.168.0.0/16 is the private subnet assigned for Pods)

Upvotes: 16

Related Questions