Cannot get mTLS working on a Google Cloud Load Balancer using PrivateCA

Question

I am trying to get mTLS working on a global HTTPS load balancer in Google Cloud, but the client connections are just being rejected with the error "client_cert_validation_failed".

I followed the exact instructions here and here but it appears the trust config supplied just isn't working.

The trust config looks like this:

trustStores:
- trustAnchors:
  - pemCertificate: |
      -----BEGIN CERTIFICATE-----
      MIICWzCCAgKgAwIBAgIUANxN5VlXyQgGZp4uZJi7iXnIivEwCgYIKoZIzj0EAwIw
      [...]
      SM49BAMCA0cAMEQCIBmn2170sahTzA0iBYuULNeywX+r8fX8JucglsMQNBT8AiAP
      vJwqN9I7bmTgItfpjozMIExZiSeiTn45TfEdNLNsTQ==
      -----END CERTIFICATE-----

NOTE: PEM is definitely correct.

I have a TLS policy which uses this:

gcloud beta network-security server-tls-policies describe london-ips-tls-policy --location=global 

createTime: '2023-08-22T15:59:51.030272533Z'
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/MY_PROJECT_NUM/locations/global/trustConfigs/london-ips-trust-config
name: projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy
updateTime: '2023-08-22T16:28:04.303764856Z'

I then have a target-https-proxies which references this:

# ...
serverTlsPolicy: //networksecurity.googleapis.com/projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy

However whenever I download one of the approved certificates like this one:

-----BEGIN CERTIFICATE-----
MIIDWzCCAwKgAwIBAgIUAKpLh7uR2I/q21WKu84Sg5VZadowCgYIKoZIzj0EAwIw
[...]
kC2s86eKXEhjU4t22QIgOHZaPGUzO3Zr9m1esHYn8T+cnSEZPaYaXigSxlil6cg=
-----END CERTIFICATE-----

Using the go code:

// load key/cert pair
    clientCertForTls, err := tls.LoadX509KeyPair(os.ExpandEnv(*clientCertPath), os.ExpandEnv(*privateKeyPath))
    if err != nil {
        log.Fatalf("unable to load x509 pair, %v", err)
    }

    // http client with client certs
    mTLSClient := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                ClientAuth:   tls.RequestClientCert,
                Certificates: []tls.Certificate{clientCertForTls},
            },
        },
    }

    // make request
    resp, err := mTLSClient.Get("https://OUR_URL.com/")
    if err != nil {
        log.Fatalf("unable to mtls ips, %v", err)
    }
    bodyDat, _ := io.ReadAll(resp.Body)
    log.Printf("body: %s", bodyDat)
    for k := range resp.Header {
        log.Printf("header[%s]: %s", k, resp.Header.Get(k))
    }
    if resp.StatusCode != http.StatusOK {
        log.Fatalf("failed to mtls ips, %s: %s", resp.Status)
    } else {
        log.Printf("ALL GOOD! :)")
    }

However the connection is always rejected by the server.

Using the logging query:

jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I can see the error "client_cert_validation_failed" in the rejection entry:

{
httpRequest: {
latency: "0s"
remoteIp: "REDACTED"
}
insertId: "REDACTED"
jsonPayload: {
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
backendTargetProjectNumber: "projects/MY_PROJECT_NUM"
remoteIp: "REDACTED"
statusDetails: "client_cert_validation_failed"
}
logName: "projects/MY_PROJECT_ID/logs/requests"
receiveTimestamp: "2023-08-22T17:00:55.441451401Z"
resource: {
labels: {
backend_service_name: ""
forwarding_rule_name: "lb-frontend-acme-protected"
project_id: "MY_PROJECT_ID"
target_proxy_name: ""
url_map_name: ""
zone: "global"
}
type: "http_load_balancer"
}
severity: "INFO"
timestamp: "2023-08-22T17:00:54.413317Z"
}

Why does this not work please?

If I change the clientValidationMode to ALLOW_INVALID_OR_MISSING_CLIENT_CERT it allows traffic through, so for some reason it's rejecting perfectly good client certificates which were issued by the Root CA (which is enabled).

UPDATE: I've also tried with cURL, but the connection is still reset, the same as the Go code: curl --key ~/.vipro-key.pem --cert ~/Downloads/client-cert.crt -v https://OUR_PROTECTED_ENDPOINT/

Answer 1

I believe the concern here is the leaf certificate. The leaf certificate cannot contain the timeStamping in the x509v3 Extended Key Usage which falls under the Certificate Requirement

You can check this by running this command “openssl x509 -in cert.pem -text”