CODEWITHSUNDEEP
dockerkubernetescontainersx509certificaterancher
asmoljo
asmoljo

Reputation: 71

Why Rancher container suddenly started to crash?

I am using Rancher v2.2.2 for managing k8s cluster in Azure AKS. Everything was fine for about one year. All of a sudden my Rancher container started to restart itself every 10 to 15 seconds and Rancher console was unreachable. I found this in container logs:

***Waiting for server to become available: Get https://localhost:6443/version?timeout=30s: x509: certificate has expired or is not yet valid***

Then I checked localhost certificate inside container and appears that it's just expired. Then I shifted time backward on docker host where is my rancher container and problem disappeared and my rancher was again live. But few weeks after I renewed k8s certificates on Azure and now my Rancher with shifted time refuses to connect to k8s cluster API server because it has certificate which is not yet valid.

Please can someone help me? Any ideas how to renew certificate inside Rancher container ?

Thanks

Upvotes: 0

Views: 3934

Answers (4)

bykebyn
bykebyn

Reputation: 85

After deleting, restarting does not generate the localhost certificate, so I manually generated the certificate.

openssl genrsa -out  localhost.key 4096
openssl req -new -key localhost.key -out localhost.csr -subj "/CN=localhost"
openssl x509 -req -in localhost.csr -CA client-ca.crt -CAkey client-ca.key -CAcreateserial -out localhost.crt -days 36500

Upvotes: 0

Sankar Prasanth Gadhamsetti
Sankar Prasanth Gadhamsetti

Reputation: 390

I just got the same issue here on my cluster.

Workaround: Set the system clock to a date in the past so that the certificate is not seen as expired. For me, on an Ubuntu server, that was achievable by disabling NTP and then setting the date and time manually;

sudo timedatectl set-ntp off
sudo date --set="2020-05-05 09:03:00.000"

Another way: Log in to the container and then delete the certificates

sudo docker exec -it rancher sh -c "rm /var/lib/rancher/management-state/tls/token-node.crt; rm /var/lib/rancher/management-state/tls/localhost.crt"

Use combination for better result without downtime

sudo timedatectl set-ntp off
sudo date --set="2020-07-11 09:03:00.000"
sudo docker exec -it rancher sh -c "rm /var/lib/rancher/management-state/tls/token-node.crt; rm /var/lib/rancher/management-state/tls/localhost.crt"
sudo timedatectl set-ntp on
sudo docker restart rancher

Upvotes: 0

JuanToroMarty
JuanToroMarty

Reputation: 506

I had the same problem, to fix it i had to delete the /var/lib/rancher/management-state folder inside the running container. Obviously, the first thing to be done is make a backup of the container data.

$ docker exec -ti sh -c "rm -rf /var/lib/rancher/management-state"

And then restart the container just to be sure about re creating the folder.

$ docker restart

I hope it helps

Upvotes: 0

asmoljo
asmoljo

Reputation: 71

I solved the problem by creating a new 'localhost' certificate inside the rancher container. The new certificate is valid for 10 years :)

Upvotes: 2

Related Questions