CODEWITHSUNDEEP
ubuntukubernetesiptablesk3s
Charles Guertin
Charles Guertin

Reputation: 33

k3s networking problems on Ubuntu 22.04: failed to get Kubernetes server version: Get \"https://10.43.0.1:443/version?timeout=32s\": i/o timeout

After upgrading from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS, I am currently facing issues with my k3s cluster.

For instance, the logs of the local-path-provisioner pod:

$ kubectl logs -n kube-system local-path-provisioner-6c79684f77-l4cqp
time="2022-04-28T03:27:00Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get \"https://10.43.0.1:443/version?timeout=32s\": dial tcp 10.43.0.1:443: i/o timeout"

I've tried the following actions:

$ sudo iptables -A INPUT -s 10.42.0.0/16 -d <host_ip> -j ACCEPT

Still, coredns, local-path-provisioner and metrics-server deployments won't start. When listing pods, here's the output:

$ kubectl get pods -n kube-system                                    
NAME                                      READY   STATUS             RESTARTS         AGE
cilium-64r4c                              1/1     Running            2 (18m ago)      174m
cilium-d8grw                              1/1     Running            2 (18m ago)      174m
cilium-g4gmf                              1/1     Running            2 (18m ago)      174m
cilium-h5j4h                              1/1     Running            2 (18m ago)      174m
cilium-n62nv                              1/1     Running            2 (18m ago)      174m
cilium-operator-76cff99967-6fgkv          1/1     Running            2 (18m ago)      174m
cilium-operator-76cff99967-pbr4l          1/1     Running            2 (18m ago)      174m
cilium-w4n6d                              1/1     Running            2 (18m ago)      174m
cilium-wgm7l                              1/1     Running            2 (18m ago)      174m
cilium-zqb6w                              1/1     Running            2 (18m ago)      174m
coredns-d76bd69b-bhgnl                    0/1     CrashLoopBackOff   44 (3m27s ago)   177m
hubble-relay-67f64789c7-cjzz9             0/1     CrashLoopBackOff   63 (4m15s ago)   174m
hubble-ui-794cd44b77-9vgbl                3/3     Running            6 (18m ago)      174m
local-path-provisioner-6c79684f77-l4cqp   0/1     CrashLoopBackOff   35 (3m53s ago)   177m
metrics-server-7cd5fcb6b7-v74rc           0/1     CrashLoopBackOff   42 (3m35s ago)   177m

Any help is appreciated! thanks

Upvotes: 3

Views: 3013

Answers (2)

Sam Day
Sam Day

Reputation: 1717

Since you're using Cilium, I think you might be running into this issue: https://github.com/cilium/cilium/issues/10645

The workaround is to ensure net.ipv4.conf.lxc*.rp_filter is set to 0:

echo 'net.ipv4.conf.lxc*.rp_filter = 0' | sudo tee -a /etc/sysctl.d/90-override.conf
sudo systemctl start systemd-sysctl

Upvotes: 2

kaffeemann
kaffeemann

Reputation: 1

I had the same problem and found this: https://github.com/k3s-io/k3s/issues/5443

You have to install the linux-modules-extra-raspi additionally: sudo apt install linux-modules-extra-raspi

$ kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
local-path-provisioner-84bb864455-bvnfh   1/1     Running   0          3m6s
coredns-96cc4f57d-28vkv                   1/1     Running   0          3m6s
metrics-server-ff9dbcb6c-m8cfd            1/1     Running   0          3m6s

Upvotes: 0

Related Questions