CODEWITHSUNDEEP
kubernetes
cryptoparty
cryptoparty

Reputation: 345

Problems with Kubernetes new workers nodes

I added node to my cluster via kubeadm join

All seems ok!

I see this node on my dashboard and healtz seems OK.

But if i start to deploy something i get error like whis one:

Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1229e227176dfefd55c0728b10045bff158d99430843aca44c8957c2850127f2" network for pod "mobsf-76dd9dc786-zjjlx": NetworkPlugin cni failed to set up pod "" network: Get https://[10.233.0.1]:443/api/v1/namespaces/default: dial tcp 10.233.0.1:443: i/o timeout

I use calico as plugin

OS: Centos 7 Kernel:5.3.6-1.

NODES: kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 143d v1.15.3 s001dev-ceph-node-01 Ready worker 40d v1.15.3 s001dev-ceph-node-02 Ready worker 40d v1.15.3 s001dev-ceph-node-03 Ready worker 40d v1.15.3 worker1 Ready worker 137d v1.15.3 worker2 Ready worker 137d v1.15.3 worker3 Ready worker 137d v1.15.3 Calico:

``

kubectl get all --all-namespaces | grep -i calico
kube-system   pod/calico-kube-controllers-6d57b44787-xlj89          1/1     Running             35         143d
kube-system   pod/calico-node-flhx4                                 1/1     Running             5          40d
kube-system   pod/calico-node-hhgzk                                 1/1     Running             30         143d
kube-system   pod/calico-node-qp2dr                                 1/1     Running             6          40d
kube-system   pod/calico-node-t698j                                 1/1     Running             183        40d
kube-system   pod/calico-node-tk4mp                                 1/1     Running             323        137d
kube-system   pod/calico-node-vg5qb                                 1/1     Running             7          40d
kube-system   pod/calico-node-w7zvs                                 1/1     Running             325        137d



kube-system   daemonset.apps/calico-node        7         7         7       7            7           <none>                        143d

kube-system   deployment.apps/calico-kube-controllers        1/1     1            1           143d

kube-system   replicaset.apps/calico-kube-controllers-6d57b44787       1         1         1       143d

Calico logs from first pod pod/calico-node-flhx4:

2020-02-03 06:55:38.094 [INFO][90] xdp_state.go 559: Finished XDP resync. family=4 resyncDuration=18.636208ms 2020-02-03 06:55:38.094 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=18.958571 2020-02-03 06:55:41.046 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:55:41.046 [INFO][90] ipsets.go 223: Asked to resync with the dataplane on next update. family="inet" 2020-02-03 06:55:41.046 [INFO][90] ipsets.go 306: Resyncing ipsets with dataplane. family="inet" 2020-02-03 06:55:41.050 [INFO][90] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=3.51651ms 2020-02-03 06:55:41.050 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=3.949963 2020-02-03 06:55:51.369 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:55:51.369 [INFO][90] ipsets.go 223: Asked to resync with the dataplane on next update. family="inet" 2020-02-03 06:55:51.370 [INFO][90] ipsets.go 306: Resyncing ipsets with dataplane. family="inet" 2020-02-03 06:55:51.373 [INFO][90] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=3.50255ms 2020-02-03 06:55:51.373 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=4.002705 2020-02-03 06:56:00.293 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:00.293 [INFO][90] table.go 740: Invalidating dataplane cache ipVersion=0x4 reason="refresh timer" table="nat" 2020-02-03 06:56:00.295 [INFO][90] table.go 460: Loading current iptables state and checking it is correct. ipVersion=0x4 table="nat" 2020-02-03 06:56:00.305 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=11.403867 2020-02-03 06:56:01.433 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:01.433 [INFO][90] ipsets.go 223: Asked to resync with the dataplane on next update. family="inet" 2020-02-03 06:56:01.433 [INFO][90] ipsets.go 306: Resyncing ipsets with dataplane. family="inet" 2020-02-03 06:56:01.437 [INFO][90] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=3.505643ms 2020-02-03 06:56:01.437 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=3.87718 2020-02-03 06:56:10.788 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:10.788 [INFO][90] table.go 740: Invalidating dataplane cache ipVersion=0x4 reason="refresh timer" table="raw" 2020-02-03 06:56:10.790 [INFO][90] table.go 460: Loading current iptables state and checking it is correct. ipVersion=0x4 table="raw" 2020-02-03 06:56:10.793 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=5.292443 2020-02-03 06:56:12.341 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:12.342 [INFO][90] ipsets.go 223: Asked to resync with the dataplane on next update. family="inet" 2020-02-03 06:56:12.342 [INFO][90] ipsets.go 306: Resyncing ipsets with dataplane. family="inet" 2020-02-03 06:56:12.344 [INFO][90] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=1.878346ms 2020-02-03 06:56:12.344 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=2.1785340000000004 2020-02-03 06:56:21.277 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:21.277 [INFO][90] table.go 740: Invalidating dataplane cache ipVersion=0x4 reason="refresh timer" table="filter" 2020-02-03 06:56:21.281 [INFO][90] table.go 460: Loading current iptables state and checking it is correct. ipVersion=0x4 table="filter" 2020-02-03 06:56:21.286 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=8.782817 2020-02-03 06:56:22.914 [INFO][90] int_dataplane.go 907: Applying dataplane updates 2020-02-03 06:56:22.914 [INFO][90] ipsets.go 223: Asked to resync with the dataplane on next update. family="inet" 2020-02-03 06:56:22.914 [INFO][90] ipsets.go 306: Resyncing ipsets with dataplane. family="inet" 2020-02-03 06:56:22.917 [INFO][90] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=2.961011ms 2020-02-03 06:56:22.917 [INFO][90] int_dataplane.go 921: Finished applying updates to dataplane. msecToApply=3.528314

From nodes worker1 and worker2 i can ping 10.233.0.1 (API), but cant at new node worker3: ping 10.233.0.1 PING 10.233.0.1 (10.233.0.1) 56(84) bytes of data. From 10.254.0.4 icmp_seq=1 Time to live exceeded From 10.254.0.4 icmp_seq=2 Time to live exceeded

I also noticed the absence calico network interfaces on promlem node

Upvotes: 2

Views: 901

Answers (1)

cryptoparty
cryptoparty

Reputation: 345

I fixed this issue! Problems was that nginx-proxy pod on problems node was in pending statyus

Upvotes: 3

Related Questions