CODEWITHSUNDEEP
kubernetes-helmamazon-ekshashicorp-vaultamazon-kmsvault
Fabio Formosa
Fabio Formosa

Reputation: 1084

Vault on EKS with KMS auto-unseal: "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

I've installed Vault 1.9.1 using the Vault Helm Chart ver 0.18.0 in my AWS EKS cluster with kubernetes 1.21.0, I'm not able to init it typing the command:

kubectl --namespace=vault exec vault-0 -- vault operator init

I get the error

Error initializing: Put "http://127.0.0.1:8200/v1/sys/init": dial tcp 127.0.0.1:8200: 
connect: connection refused

The pod is running but not in READY status, the readiness probe fails due to:

Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

This is my chart values:

vault:
  injector:
    enabled: false
  csi:
    enabled: true
  server:
    enabled: true
    extraVolumes:
      - name: vault-storage-config
        type: secret
    extraArgs: -config=/vault/userconfig/vault-storage-config/config.hcl
    ha:
      enabled: true
      replicas: 3

and the config.hcl

ui = true

storage "postgresql" {
  connection_url = "postgres://<user>:<pwd>@<rds.url>/vault"
  table="vault_kv_store"
  ha_enabled="true"
  ha_table="vault_ha_locks"
}

service_registration "kubernetes" {}

seal "awskms" {
  kms_key_id = <my_kms_key_id>
}

I've enabled the auto-unseal feature leveraging the integration with AWS KMS. I've already checked that the EKS worker nodes are able to reach the postgres RDS instance and able to call the AWS KMS service, they are granted to

{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:DescribeKey",
            "ec2:DescribeInstances"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
    ]
}

any help?

Upvotes: 0

Views: 611

Answers (1)

Fabio Formosa
Fabio Formosa

Reputation: 1084

Found the solution in my own. I was focusing on the auto-unsealing feature supposing an issue on permissions, actually later I found out it was an issue related to the postgres data storage.
Due to the restarting, I lost the logs that later I was able to get typing

kubectl logs vault-0 -n mynamespace --previous

So I noticed the error

Error initializing storage of type postgresql: 
failed to check for native upsert: dial tcp <rds_ip_instance>:5342: connect: connection timed

Early I checked the reachability of the rds instance using psql from the host, but I didn't notice I misconfigured the postgres port 5343 instead 5432.

Basically, I has been victim of the jumbled word effect that has driven me crazy for 2 long days!

Upvotes: 2

Related Questions