I can't get `golang.org/x/crypto/acme/autocert` to work (for gRPC), I get an `acme_account+key` file and no X509 cert

Question

UPDATE After much debugging, I uncovered Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority and suspect (!?) this results from the recent expiration of Let's Encrypt's root cert.

I accept that "This package is a work in progress and makes no API stability promises." but, if it no longer works (and it's much more likely that my code|deployment is at issue), then perhaps the repo can be marked e.g. Here be dragons.

The code results in an acme_account+key (EC PRIVATE KEY) but no certs I'm challenged to get autocert to disclose (log) its magic in order to understand where I'm going wrong.

The code is essentially the repo's Manager example with input from this answer. I assume that GetCertificate blocks on the completion of the ACME flow.

Code:

package main

import (
    "crypto/tls"
    "flag"
    "fmt"
    "log"
    "net"
    "net/http"

    "golang.org/x/crypto/acme/autocert"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/health"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

const (
    email string = "my@email.com"

var (
    host = flag.String("host", "foo.example.org", "Fully-qualified domain name")
    port = flag.Uint("port", 443, "gRPC service port")
    path = flag.String("path", "", "Folder location for certificate")
)

func main() {
    flag.Parse()

    if *host == "" {
        log.Fatal("Flag --host is required")
    }
    log.Printf("Host: %s", *host)
    log.Printf("Port: %d", *port)

    if *path == "" {
        log.Fatal("Flag --path is required")
    }
    log.Printf("Path: %s", *path)

    addr := fmt.Sprintf(":%d", *port)

    lis, err := net.Listen("tcp", addr)
    if err != nil {
        log.Fatalf("failed to listen: %s", err)
    }

    m := &autocert.Manager{
        Cache:      autocert.DirCache(*path),
        Prompt:     autocert.AcceptTOS,
        HostPolicy: autocert.HostWhitelist(*host),
        Email:      email,
    }

    go func() {
        log.Println("Starting HTTP server w/ autocert handler")
        if err := http.ListenAndServe(":http", m.HTTPHandler(nil)); err != nil {
            log.Fatalf("HTTP failure\n%s", err)
        }
    }()

    tlsConfig := &tls.Config{
        ClientAuth: tls.RequireAndVerifyClientCert,
        GetCertificate: func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
            cert, err := m.GetCertificate(hello)
            if err != nil {
                log.Fatalf("GetCertificate\n%s", err)
            }
            return cert, err
        },
    }

    opts := grpc.Creds(credentials.NewTLS(tlsConfig))
    server := grpc.NewServer(opts)

    healthcheck := health.NewServer()
    healthpb.RegisterHealthServer(server, healthcheck)

    log.Println("Starting gRPC server")
    if err := server.Serve(lis); err != nil {
        log.Fatalf("gRPC failure\n%s", err)
    }
}

I'm deploying to a (Google Compute Engine) Container VM, the equivalent docker command is:

docker run \
--name=autocert \
--detach \
--net=host \
--volume=/tmp/certs:/certs \
${IMAGE} \
--host=${HOST} \
--port=${PORT} \
--path=/certs

And container logs:

2021/11/25 17:30:00 Host: [HOST]
2021/11/25 17:30:00 Port: 443
2021/11/25 17:30:00 Path: /certs
2021/11/25 17:30:00 Starting gRPC server
2021/11/25 17:30:00 Starting HTTP server

The host's /tmp/certs directory receives acme_account+key (which I've struggled to find explained by Google) but suspect (!?) is the initial phase of Domain Validation. It contains a private key (BEGIN EC PRIVATE KEY).

Even after some time with the server running, no further files are persisted.

I receive no emails from Let's Encrypt at the configured email address.

Unfortunately, while easy to use, autocert produces little logging and I've been unable to determine whether I can log the ACME flow that's (hopefully) taking place.

Since adding the anonymous function for GetCertificate, acme_account+key is no longer created (I removed the previous file to check whether it's recreated) and so I'm unable to gather any logging from it but the function is never invoked. Either this is because my anonymous function is incorrect or because I've exceeded requests against the ACME endpoint. Removing the function and reverting to m.GetCertificate does not result in recreation of acme_account+key so I'm at a loss.

The autocert Manager type documents an *acme.Client field which I'm not setting. The comment describes "if the Client.Key is nil, a new ECDSA P-256 key is generated" which is perhaps what I'm experiencing but it doesn't explain what I should do about it. Should I set this value to the content of acme_account+key?:

UPDATE I tried decoding the private key, creating a crypto.Signer and passing this in &acme.Client{Key: key} but it made no evident difference

// Client is used to perform low-level operations, such as account registration
// and requesting new certificates.
//
// If Client is nil, a zero-value acme.Client is used with DefaultACMEDirectory
// as the directory endpoint.
// If the Client.Key is nil, a new ECDSA P-256 key is generated and,
// if Cache is not nil, stored in cache.
//
// Mutating the field after the first call of GetCertificate method will have no effect.
Client *acme.Client

Evidently, I'm using this incorrectly. I'm not receiving a cert from Let's Encrypt and so I'm unable to get a cert from the endpoint and unable to invoke the gRPC endpoint:

openssl s_client -showcerts -connect ${HOST}:${PORT}

grpcurl \
-proto health.proto \
${HOST}:${PORT} \
grpc.health.v1.Health/Check
Failed to dial target host "${HOST}:${PORT}": remote error: tls: internal error

Guidance would be appreciated.

Answer 1

🤦‍♂️

Ugh :-(

I'd started using SCRATCH and hadn't copied the CA certificates

Once the container had CA certs, everything worked almost flawlessly.

I continue to have problems trying to use:

tlsConfig := &tls.Config{
  ClientAuth: tls.RequireAndVerifyClientCert,
  GetCertificate: m.GetCertificate
}

And am using m.TLSConfig()

So, autocert works like a treat (though it's difficult to debug self-inflicted errors 😊)