CODEWITHSUNDEEP
nginxkuberneteslets-encryptcert-managerk3s
Sebastiano Gazzola
Sebastiano Gazzola

Reputation: 69

Can't complete HTTP challenge for letsencrypt on Kubernetes

I have a k3s cluster and I'm trying to configure it to get a SSL certificate from let's encrypt. I have followed many guides, and I think I'm really near to manage it, but the problem is that the Challenge object in Kubernetes reports this error:

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://devstore.XXXXXXX.com/.well-known/acme-challenge/kVVHaQaaGU7kbYqnt8v7LZGaQvWs54OHEe2WwI_MOgk': Get "http://devstore.XXXXXXX.com/.well-known/acme-challenge/kVVHaQaaGU7kbYqnt8v7LZGaQvWs54OHEe2WwI_MOgk": dial tcp: lookup devstore.XXXXXXX.com on 10.43.0.10:53: no such host

It seems that the in some way cert manager is trying to resolve my public DNS name internally, and is not managing to do it, so the challenge is not working. Can you help me on that, I googled it but I cannot find a solution for it...

Thank you

Upvotes: 1

Views: 2294

Answers (4)

markus
markus

Reputation: 91

In my case it was a configuration problem:

I misunderstood the k3s --cluster-domain option (which feeds into kubelet's --cluster-domain option) to mean the public domain of the cluster. All the k3s documentation offers is "Cluster Domain".

But --cluster-domain must be a private, cluster-internal domain like the default "cluster.local".

By assigning the public domain, I basically removed it from external DNS lookup, i.e. coredns refused to DNS-recurse. Hence the HTTP challenge self check did not work.

Removing the --cluster-domain option (i.e. letting it default to "cluster.local"), and rebooting the cluster solved the issue.

Upvotes: 0

Sebastiano Gazzola
Sebastiano Gazzola

Reputation: 69

Thank you DarthHTTP, I finally manage to make it work! The problem was, as I mentioned on the comment, that the firewall was not routing correctly the HTTP request using the public IP from the private network side. I solved configuring an internal DNS server that is resolving the name with the private IP address of the K3S node, and using that server as the DNS server for the K3S node. Eventually my HTTP web app got a valid let's encrypt certificate!

Upvotes: 0

Janis Kirsteins
Janis Kirsteins

Reputation: 2138

This can be caused by multiple different reasons. If you find that it is a transient issue (or possibly if you have misconfigured coredns before), you might want to double-check your coredns configmap (in the kube-system namespace).

E.g. you could remove/reduce caching, or point to different DNS nameservers.

Here's a description of the issue, where a switch to Google DNS + cache removal helped clear the issue.

Upvotes: 0

DarthHTTP
DarthHTTP

Reputation: 406

It is probable that the DNS record for the domain you want the certificate does not exist. If if does, and you are using a split horizon DNS config (hijacking the .com domain in your local network) make sure it points out to your public ip (e.g. your home gateway)

[Edit]

Also, you have to figure out LE getting to your cluster in the network, so port-forward 80/443 to your cluster's IPs.

You can get away with it because k3s will default to cluster traffic policy in the load balancer

Upvotes: 2

Related Questions