CODEWITHSUNDEEP
amazon-web-serviceskubernetesamazon-eks
Yechan Kim
Yechan Kim

Reputation: 66

AWS EKS is giving x509 error after upgrade it to kubernetes v1.11.5

I upgraded my EKS cluster to v1.11.5 from v.1.10.3, but it's giving me x509: cannot validate certificate for <WORKER_IP> because it doesn't contain any IP SANs error when i try to get logs or helm ls on it. Other commands like kubectl get nodes are working fine.

For the upgrade, I clicked "Upgrade cluster" button on the web console, and modified cloudformation template for workers to use latest AWS provided ami (ami-0a9006fb385703b54). read this guide from AWS, including this.

My cluster was completely unuseable so I rollbacked my worker nodes with old ami (ami-0c7a4976cb6fafd3a) for now, and the error is gone.

I'm really not sure that what i missed. Anyone experiencing similar issue with me? I need help.

Thanks.

Upvotes: 1

Views: 932

Answers (2)

Shahar Hamuzim Rajuan
Shahar Hamuzim Rajuan

Reputation: 6129

I am using k8s 1.18 on EKS deployed via eksctl version 0.30.0, I had the same issue when trying to connect to pods on a specific instance in the cluster. Got this error:

2021-01-12T13:41:10.409327444Z Error from server: error dialing backend: x509: cannot validate certificate for 10.000.12.04 because it doesn't contain any IP SANs

I logged in to the instance:

sudo ls -la /var/lib/kubelet/pki
total 12
drwxr-xr-x 2 root root  124 Jan 10 12:48 .
drwxr-xr-x 8 root root  154 Jan 10 11:38 ..
-rw-r--r-- 1 root root 2173 Jan 10 12:48 kubelet.crt
-rw------- 1 root root 1679 Jan 10 12:48 kubelet.key
-rw------- 1 root root 1232 Jan 10 11:38 kubelet-server-2021-01-10-11-38-05.pem
lrwxrwxrwx 1 root root   59 Jan 10 11:38 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2021-01-10-11-38-05.pem

The server certificate was not using the correct key.

I copied the server certificate from the original certificate created while the instance was bootstrapped ( kubelet-server-2021-01-10-11-38-05.pem) in to kubelet.crt. And also copied private key should go to kubelet.key

Then restarted the kubelet:

sudo service kubelet restart

And everything was working fine.

But I think the best way is to just delete the instance from the ASG and create a new one if possible.

Upvotes: 0

Harsh Manvar
Harsh Manvar

Reputation: 30083

x509 is token authentication and authorisation way in kubernetes may be some that type of problem is there auth related.

Upvotes: 1

Related Questions