Pods are remain ContainerCreating. networkPlugin cni failed to set up pod netplugin failed with no error message

Question

I am trying to create a deployment on a K8s cluster with one master and two worker nodes. The cluster is running on 3 AWS EC2 instances and using a weave network.

[root@master ~]# export kubever=$(kubectl version | base64 | tr -d '\n')
[root@master ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"

[root@master ~]# kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
nginx-f89759699-7vscl   0/1     ContainerCreating   0          36m

kubectl describe pod nginx -n mounika

Name:           nginx-f89759699-7vscl
Namespace:      mounika
Priority:       0
Node:           worker2/172.31.57.238
Start Time:     Sun, 31 May 2020 01:58:15 +0000
Labels:         app=nginx
                pod-template-hash=f89759699
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/nginx-f89759699
Containers:
  nginx:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p5h57 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-p5h57:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p5h57
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               30m                default-scheduler  Successfully assigned mounika/nginx-f89759699-7vscl to worker2
  Warning  FailedMount             30m                kubelet, worker2   MountVolume.SetUp failed for volume "default-token-p5h57" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedCreatePodSandBox  26m                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f2e3044010c65e0f7197ef73b72fee041b6a180d14b4a527fa1df14e30fbc658" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  22m                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d3c25be771a95755230c5941be0dba193d948e089e8d2f280597d5dd45400053" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  19m                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "012db3142ca5c763315e44178255003d8429baf1f6cd970c5730735fcad97121" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  15m                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "73ec714e5aebf0336551805095c9560d9da61ca9af31c70d9f95a54b7841bc29" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  11m                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5be6e05842ddbe3d02f107b90db17c0cf1a7975ca151a826ba3fe4a7c804be45" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  8m17s              kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ba2e1f771f08352eaf254e2e708cc44516217b388ae2675380e42eed28839576" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Warning  FailedCreatePodSandBox  4m36s              kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "02a2834f8ef92152ce5141cc99f8e92bd0496a6b0b91fd20df21b020b4f8160d" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
  Normal   SandboxChanged          55s (x8 over 26m)  kubelet, worker2   Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  55s                kubelet, worker2   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e91972b0b5d2d056ff8e354973bf4d7d14899ae730c285475ff2beb12dd9d25b" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message ```

Manually pulled the image in worker nodes too there is no network issue.

[root@worker2 ~]# docker pull nginx Using default tag: latest Status: Downloaded newer image for docker.io/nginx:latest

3 days onwards searching for the solution but didn't get.

[root@master ~]# kubectl get pods -n kube-system
NAME                             READY   STATUS      RESTARTS   AGE
coredns-66bff467f8-64fdb         0/1     Completed   0          165m
coredns-66bff467f8-l6cfn         0/1     Completed   0          165m
etcd-master                      1/1     Running     1          165m
kube-apiserver-master            1/1     Running     1          165m
kube-controller-manager-master   1/1     Running     1          165m
kube-proxy-5nnjp                 1/1     Running     1          165m
kube-proxy-c9rdt                 1/1     Running     1          157m
kube-proxy-pm7rv                 1/1     Running     1          158m
kube-scheduler-master            1/1     Running     1          165m
weave-net-hhx92                  2/2     Running     3          158m
weave-net-rt6b9                  2/2     Running     4          157m
weave-net-s8vxz                  2/2     Running     2          161m
[root@master ~]#

[root@master ~]# export kubever=$(kubectl version | base64 | tr -d '\n')
[root@master ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"

kubelet logs of worker2:

May 31 04:42:18 worker2 kubelet[7469]: E0531 04:42:18.523106    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:42:28 worker2 kubelet[7469]: E0531 04:42:28.533263    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:42:28 worker2 kubelet[7469]: E0531 04:42:28.533765    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:42:33 worker2 kubelet[7469]: E0531 04:42:33.277736    7469 cni.go:364] Error adding mounika_nginx-f89759699-7vscl/4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59 to network weave-net/weave: netplugin failed with no error message
May 31 04:42:33 worker2 kubelet[7469]: weave-cni: error removing interface "eth0": Link not found
May 31 04:42:33 worker2 kubelet[7469]: weave-cni: Delete: no addresses for 4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59
May 31 04:42:33 worker2 kubelet[7469]: E0531 04:42:33.382722    7469 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
May 31 04:42:33 worker2 kubelet[7469]: E0531 04:42:33.382778    7469 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "nginx-f89759699-7vscl_mounika(5c67bff1-307e-4862-8435-1e5064eed868)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
May 31 04:42:33 worker2 kubelet[7469]: E0531 04:42:33.382796    7469 kuberuntime_manager.go:727] createPodSandbox for pod "nginx-f89759699-7vscl_mounika(5c67bff1-307e-4862-8435-1e5064eed868)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59" network for pod "nginx-f89759699-7vscl": networkPlugin cni failed to set up pod "nginx-f89759699-7vscl_mounika" network: netplugin failed with no error message
May 31 04:42:33 worker2 kubelet[7469]: E0531 04:42:33.382863    7469 pod_workers.go:191] Error syncing pod 5c67bff1-307e-4862-8435-1e5064eed868 ("nginx-f89759699-7vscl_mounika(5c67bff1-307e-4862-8435-1e5064eed868)"), skipping: failed to "CreatePodSandbox" for "nginx-f89759699-7vscl_mounika(5c67bff1-307e-4862-8435-1e5064eed868)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-f89759699-7vscl_mounika(5c67bff1-307e-4862-8435-1e5064eed868)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59\" network for pod \"nginx-f89759699-7vscl\": networkPlugin cni failed to set up pod \"nginx-f89759699-7vscl_mounika\" network: netplugin failed with no error message"
May 31 04:42:33 worker2 kubelet[7469]: W0531 04:42:33.810426    7469 docker_sandbox.go:400] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "nginx-f89759699-7vscl_mounika": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59"
May 31 04:42:33 worker2 kubelet[7469]: W0531 04:42:33.811411    7469 pod_container_deletor.go:77] Container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59" not found in pod's containers
May 31 04:42:33 worker2 kubelet[7469]: W0531 04:42:33.813100    7469 cni.go:331] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59"
May 31 04:42:33 worker2 kubelet[7469]: weave-cni: Delete: no addresses for 4db7e6341952ff08d12ef14c4cdffb7c937e2e4840b2226a414e1d1079f8fc59
May 31 04:42:38 worker2 kubelet[7469]: E0531 04:42:38.547381    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:42:38 worker2 kubelet[7469]: E0531 04:42:38.547405    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:42:48 worker2 kubelet[7469]: E0531 04:42:48.558802    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:42:48 worker2 kubelet[7469]: E0531 04:42:48.558828    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:42:58 worker2 kubelet[7469]: E0531 04:42:58.569621    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:42:58 worker2 kubelet[7469]: E0531 04:42:58.569648    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:43:08 worker2 kubelet[7469]: E0531 04:43:08.580443    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:43:08 worker2 kubelet[7469]: E0531 04:43:08.580469    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:43:18 worker2 kubelet[7469]: E0531 04:43:18.590888    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:43:18 worker2 kubelet[7469]: E0531 04:43:18.590914    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
May 31 04:43:28 worker2 kubelet[7469]: E0531 04:43:28.605194    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
May 31 04:43:28 worker2 kubelet[7469]: E0531 04:43:28.605218    7469 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"
[root@worker2 ~]#

[root@master ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master    Ready    master   4h11m   v1.18.3
worker1   Ready    <none>   4h3m    v1.18.3
worker2   Ready    <none>   4h3m    v1.18.3
[root@master ~]#

Answer 1

allow the CNI 6783 and 6784 ports in the nodes security group