CODEWITHSUNDEEP
google-kubernetes-enginehashicorp-vaultgoogle-cloud-kms
GuiFP
GuiFP

Reputation: 81

can't map (inside gke) Vault to Google KMS

We want to deploy Hashicorp Vault (fork by Banzai Cloud) inside our GKE cluster and then map it to Cloud KMS / Firestore. We did it already on EKS / S3 / AWS KMS and it works fine.

However, on GKE, vault pods are crashlooping with following error message :

2021-02-09T11:36:46.455Z [WARN]  storage migration check error: error="failed to read value for "core/migration": googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>UserProjectAccountProblem</Code><Message>User project billing account not in good standing.</Message><Details>The billing account for the owning project is disabled in state closed</Details></Error>"

Does that ring any bell to anyone? We talked to the billing support to confirm we do not have any billing issue. This error message is not true and we don't have any clue what's going on.

One of our try was using Helm with following commands (otherwise we would directly use the vault operator) :

❯ kubectl create secret -n vault generic gcp-sa-vault-json --from-literal=GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/gcp-sa-vault.json --from-file=gcp-sa-vault.json=./gcp-sa-vault.json

❯ helm install vault banzaicloud-stable/vault -n vault \
--set "vault.customSecrets[0].secretName=gcp-sa-vault-json" \
--set "vault.customSecrets[0].mountPath=/etc/gcp" \
--set "vault.config.storage.gcs.bucket=vault-ha" \
--set "vault.config.seal.gcpckms.project=our-projectID" \
--set "vault.config.seal.gcpckms.region=europe-west1" \
--set "vault.config.seal.gcpckms.key_ring=vault" \
--set "vault.config.seal.gcpckms.crypto_key=vault-unsealer" \
--set "unsealer.args[0]=--mode" \
--set "unsealer.args[1]=google-cloud-kms-gcs" \
--set "unsealer.args[2]=--google-cloud-kms-key-ring" \
--set "unsealer.args[3]=vault" \
--set "unsealer.args[4]=--google-cloud-kms-crypto-key" \
--set "unsealer.args[5]=vault-unsealer" \
--set "unsealer.args[6]=--google-cloud-kms-location" \
--set "unsealer.args[7]=europe-west1" \
--set "unsealer.args[8]=--google-cloud-kms-project" \
--set "unsealer.args[9]=our-projectID" \
--set "unsealer.args[10]=--google-cloud-storage-bucket" \
--set "unsealer.args[11]=vault-ha"

Also, the google service account used to access cloud-kms have the right permissions to my knowledge :

Upvotes: 1

Views: 350

Answers (2)

GuiFP
GuiFP

Reputation: 81

After reading @Tim Dierks comment, I tried re-creating the firestore bucket (I had deleted it in case Vault would create it automatically since it has Storage Admin permissions) and it started working.

Upvotes: 1

David C
David C

Reputation: 403

I think on your case you need to create a support ticket on GCP Support, because is a very specific issue within you GCP-GKE-Hashicorp-Vault. Are you following some guide? I found the Hashicorp documentation, or you are following another guide which could provide more context?

Upvotes: 0

Related Questions