How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)

Question

Requesting for some help on : How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)

I'm trying to build a POC, for Vault and consul and got some questions.

1. Deployed Azure managed app using - https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-deploy

2. Installed consul agent on AKS with the steps in the https://learn.hashicorp.com/tutorials/consul/hashicorp-consul-service-aks?in=consul/hcs-azure Consul helm chart : https://github.com/hashicorp/consul-helm

3. Installed vault via helm chart: https://github.com/hashicorp/vault-helm

Kubernetes services and pods for consul.

~$kubectl get svc -n consul                                            
NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
consul-connect-injector-svc   ClusterIP   10.0.252.97   <none>        443/TCP   2d13h
consul-controller-webhook     ClusterIP   10.0.169.80   <none>        443/TCP   2d13h

~$kubectl get pods -n consul
NAME                                                         READY   STATUS    RESTARTS   AGE
consul-27j4j                                                 1/1     Running   0          2d13h
consul-connect-injector-webhook-deployment-9454b8d68-778rd   1/1     Running   0          2d13h
consul-controller-7857456f99-mhzpw                           1/1     Running   1          2d13h
consul-lkhpl                                                 1/1     Running   0          2d13h
consul-webhook-cert-manager-cfbb689f7-fgtlw                  1/1     Running   0          2d13h
consul-zf989                                                 1/1     Running   0          2d13h

vault config as below:

ui:
  enabled: true
  serviceType: LoadBalancer
server:
  ingress:
    enabled: true
    extraPaths:
      - path: /
        backend:
          serviceName: vault-ui
          servicePort: 8200
    hosts:
      - host: vault.something_masked.com
  ha:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "consul" {
        path = "vault/"
        scheme = "https"
        address = "HOST_IP:8500"
      }

Vault pods

kubectl get pods -n vault                             
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                0/1     Running   0          7m14s
vault-1                                0/1     Running   0          7m11s
vault-2                                0/1     Running   0          7m11s
vault-agent-injector-cbbb6f4df-rmbd7   1/1     Running   0          7m22s

ERROR : Vault is unable to make communication with consul agent.

Logs for vault-0 pod

kubectl logs vault-0 -n vault

WARNING! Unable to read storage migration status.
2021-06-27T08:37:17.801Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-27T08:37:18.824Z [WARN]  storage migration check error: error="Get "https://10.54.0.206:8500/v1/kv/vault/core/migration": dial tcp 10.54.0.206:8500: connect: connection refused"

Logs for vault-agent-injector pod

 kubectl logs vault-agent-injector-cbbb6f4df-rmbd7 -n vault    
2021-06-27T08:37:09.189Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2021-06-27T08:37:09.218Z [INFO]  handler.auto-tls: Generated CA
2021-06-27T08:37:09.219Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-27T08:37:18.252Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-06-27T08:37:18.452Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s

Any suggestions or advice in the vault configuration if I have missed something ??

Thank you in advance.

Regards Pooja

Answer 1

I am using consul azure managed app server and i have installed consul agent on aks.

kubectl get svc -n consul
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-connect-injector-svc ClusterIP 10.0.252.97 <none> 443/TCP 3d13h
consul-controller-webhook ClusterIP 10.0.169.80 <none> 443/TCP 3d13h

as i am using consul agent i do not see consul-server running.

Helm chart config for consul

global:
  enabled: false
  name: consul
  datacenter: dc1
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: XXX-sandbox-managed-app-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: XXX-sandbox-managed-app-hcs
    secretKey: gossipEncryptionKey
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: XXX-sandbox-managed-app-hcs
      secretKey: caCert
externalServers:
  enabled: true
  hosts:
    ['XXX.az.hashicorp.cloud']
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: https://XXX.uksouth.azmk8s.io:443
client:
  enabled: true
  # If you are using Kubenet in your AKS cluster (the default network),
  # uncomment the line below.
  # exposeGossipPorts: true
  join:
    ['XXX.az.hashicorp.cloud']
connectInject:
  enabled: true
controller:
  enabled: true

Helm chart config for vault

ui:
  enabled: true
  serviceType: LoadBalancer

server:
  ingress:
    enabled: true
    extraPaths:
      - path: /
        backend:
          serviceName: vault-ui
          servicePort: 8200
    hosts:
      - host: something.com
  ha:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "consul" {
        path = "vault/"
        scheme = "https"
        address = "HOST_IP:8500"
      }

Error in vault pod which is unable to connect to consul agent.

kubectl logs vault-0 -n vault

WARNING! Unable to read storage migration status.
2021-06-28T08:13:13.041Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-28T08:13:13.042Z [WARN]  storage migration check error: error="Get "https://127.0.0.1:8500/v1/kv/vault/core/migration": dial tcp 127.0.0.1:8500: connect: connection refused"

i am not sure if some configuration is missed in consul helm chart as i do not see any service running on port 8500 in consul namespace.

Any suggestion would be much appreciated.

Thanks, pooja

Answer 2

There are many points to debug the issue and the problem can be anywhere

the major issue I am seeing right now is your vault pods are running but not in a Ready state.

You have to unseal the vault from seal status.

Read more about the Unseal : https://learn.hashicorp.com/tutorials/vault/ha-with-consul#step-5-start-vault-and-verify-its-state

also how your consul is iteracting with vault ? Using LB, Ingress, service name ?