CODEWITHSUNDEEP
dockerkubernetesweave
taibc
taibc

Reputation: 923

Can not ping to pod's ip of worker node in kubernetes

My cluster includes: 1 master and 2 worker nodes. I created a pod using deployment yaml. The pod running successfully on the worker node 1, I can ping the pod's ip on worker nodes but I can't ping the ip address of the pod on the master. I tried to disable firewarlld, restart docker but not successfully. Please see my commands

[root@k8s-master ~]# kubectl get pods -o wide | grep qldv
qldv-liberty-8499dfcf67-55njr   1/1     Running             0          6m42s   10.40.0.2    worker-node1   <none>           <none>

[root@k8s-master ~]# ping 10.40.0.2
PING 10.40.0.2 (10.40.0.2) 56(84) bytes of data.
From 10.32.0.1 icmp_seq=1 Destination Host Unreachable
From 10.32.0.1 icmp_seq=2 Destination Host Unreachable
From 10.32.0.1 icmp_seq=3 Destination Host Unreachable

[root@k8s-master ~]# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
k8s-master     Ready    master   43d   v1.15.0
worker-node1   Ready    <none>   42d   v1.15.0
worker-node2   Ready    <none>   42d   v1.15.0


[root@k8s-master ~]# kubectl describe pod qldv-liberty-8499dfcf67-55njr
Name:           qldv-liberty-8499dfcf67-55njr
Namespace:      default
Priority:       0
Node:           worker-node1/192.168.142.130
Start Time:     Sat, 17 Aug 2019 20:05:57 +0700
Labels:         app=qldv-liberty
                pod-template-hash=8499dfcf67
Annotations:    <none>
Status:         Running
IP:             10.40.0.2
Controlled By:  ReplicaSet/qldv-liberty-8499dfcf67
Containers:
  qldv-liberty:
    Container ID:   docker://03636fb62d4cca0e41f4ad9f5a94b50cf371089ab5a0813ed802d02f4ac4b07a
    Image:          qldv-liberty
    Image ID:       docker://sha256:bd0d7ce1c07da5b9d398131b17da7a6931a9b7ae0673d19a6ec0c409416afc69
    Port:           9080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sat, 17 Aug 2019 20:06:23 +0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vtphv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-vtphv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vtphv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                   Message
  ----    ------     ----  ----                   -------
  Normal  Scheduled  119s  default-scheduler      Successfully assigned default/qldv-liberty-8499dfcf67-55njr to worker-node1
  Normal  Pulled     96s   kubelet, worker-node1  Container image "qldv-liberty" already present on machine
  Normal  Created    95s   kubelet, worker-node1  Created container qldv-liberty
  Normal  Started    91s   kubelet, worker-node1  Started container qldv-liberty

I have another app, it also has a pod that running on the worker node 1, and I can ping the pod's ip from master. But I don't know why it is impossible with above case. Please help me !

Upvotes: 3

Views: 5325

Answers (1)

VAS
VAS

Reputation: 9031

I doubt that the cluster still exists, therefore I'd better share some troubleshooting tips:

  1. Check status of all control plane components and node status. Ensure kube-proxy and network addon (flannel/calico/waive/etc) Pods exist on each node and in Ready state.
kubectl get deployments,daemonsets,pods,svc -A -o wide

        There are several requirements for Kubernetes cluster, and it worth to check if they are satisfied.

        Some useful information could be found in the control-plane component logs using

kubectl logs kube-component-name-pod -n kube-system

        or kubelet logs using

journalctl -u kubelet

  1. It's better to use well known images like nginx or mendhak/http-https-echo. They could be configured to listen any desired port and provide detailed information about requests in logs or in HTTP reply. It helps to exclude application/image related issues.

  2. Check connectivity to Pod IP and Service ClusterIP within the same node first.
    If worker node OS doesn't have necessary tools for troubleshooting (e.g container optimized images or coreOS), Pod with Ubuntu or Busybox image can be used for that. Creating Deployment or DaemonSet could help to schedule it on all nodes. Note that firewall or network issues can block kubectl exec connections to that pods.

  3. If everything works fine within the same node, but connection to the Pod can't be established from another node it worth to check network addon status and nodes firewall configuration. Native Linux firewall helpers can interfere with iptables set of rules created by kube-proxy and block the connection.

        Clusters created in public clouds may require additional routing, peering, cloud firewall or security groups configuration to allow full IPIP connectivity between cluster nodes, especially if they are created in different VPCs.

  1. The next thing that worth to check is coredns/kube-dns health. They suppose to resolve to correct IP address cluster Services' names like servicename.namespacename.svc.cluster.local if requested using their pod IP addresses or kube-dns Service (it usually has IP address 10.96.0.10 in default kubeadm cluster configuration).

Solution for each problem could be found in another answers on StackExchange sites. Official documentation is another great source of information and also contains good examples.

Upvotes: 0

Related Questions