CODEWITHSUNDEEP
kubernetesweave
S Andrew
S Andrew

Reputation: 7208

Readiness probe failed error in weave kubernetes

I have cluster where I have 10 worker nodes on Raspberry pi and master is running on Ubuntu 16.04. Everything seems to be working fine but sometime a pod running on node shows below error:

Warning FailedCreatePodSandBox 18m (x3 over 18m) kubelet, w188 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6fa511fb1d292702aa99318b785f5011307601868ff2520c542515a239924c16" network for pod "deployment-6w24f": NetworkPlugin cni failed to set up pod "deployment-6w24f_aps-namespace" network: unable to allocate IP address: Post http://127.0.0.1:6784/ip/6fa511fb1d292702aa99318b785f5011307601868ff2520c542515a239924c16: dial tcp 127.0.0.1:6784: connect: connection refused

I am using weave for networking and it shows below error on kube dashboard:

Readiness probe failed: Get http://127.0.0.1:6784/status: dial tcp 127.0.0.1:6784: connect: connection refused Back-off restarting failed container MountVolume.SetUp failed for volume "weave-net-token-txqhk" : couldn't propagate object cache: timed out waiting for the condition

The node shows this error and after sometime it starts working fine automatically. This is happening every now and then with multiple nodes. I used below commands to init the cluster and weave:

sudo kubeadm init --token-ttl=0 --apiserver-advertise-address=192.168.8.12

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Can anyone please guide me on how to resolve this.

Upvotes: 4

Views: 4658

Answers (2)

Shivaraj Navalgund
Shivaraj Navalgund

Reputation: 367

In my case, I had missed deleting /etc/cni/net.d/10-aws.conflist file on each of the nodes.

In summary, we need to follow below steps:

  1. kubectl delete ds aws-node -n kube-system
  2. delete /etc/cni/net.d/10-aws.conflist on each of the node
  3. restart kube-proxy pods
  4. Install Weave Net resources. E.g.: kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

Official Documentation to install Weave Net on AWS EKS Cluster

Upvotes: 0

Lukasz Dynowski
Lukasz Dynowski

Reputation: 13610

What worked for me was to add iptable rules with the IP range of my kube-dns service.

# kubectl get svc -n kube-system 
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   16m
[root@master kubernetes]#

Since IP of kube-dns is 10.96.0.10 then, the valid range for iptable rule would be 10.96.0.1/32

# iptables -t nat -I KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ

More about this issue can be found here.

Upvotes: 4

Related Questions