Hashicorp vault policy fail - what am I missing here?

Question

I'm trying to use gitlab-ci along with hashicorp vault to provide secrets to ci jobs via jwt auth. I seem to have everything working except the policy.

So I first enabled the jwt auth method:

vault auth enable jwt

Then created a relevant policy:

vault policy write k8s-gcp-env - <<EOF
path "gitlab/k8s-gcp-env/*" {
        capabilities = [ "read", "list" ]
}
EOF

And a role:

vault write auth/jwt/role/k8s-gcp-env - <<EOF
{
        "role_type": "jwt",
        "policies": ["k8s-gcp-env"],
        "token_explicit_max_ttl": 60,
        "bound_claims_type": "glob",
        "bound_claims": {
                "project_id": "28"
        }
}
EOF

Then configured the jwt auth method (with a custom CA):

vault write auth/jwt/config \                                                                                                                        
jwks_url="https://git.__REDACTED__/-/jwks" \
bound_issuer="git.__REDACTED__" \
jwks_ca_pem=@/Users/user/Downloads/c81a8bd1f9cf6d84c525f378ca1d3f8c30770e34.cer

This all worked fine, but my pipeline fails to read secrets:

$ export VAULT_ADDR=https://vault.__REDACTED__
$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=$CI_PROJECT_NAME jwt=$CI_JOB_JWT)"
$ echo $VAULT_TOKEN
__REDACTED__
$ vault token lookup
Key                 Value
---                 -----
accessor            __REDACTED__
creation_time       1666877529
creation_ttl        1m
display_name        __REDACTED__
entity_id           23938616-4ca5-fd51-b607-9a029476ab6d
expire_time         2022-10-27T13:33:09.410411192Z
explicit_max_ttl    1m
id                  __REDACTED__
issue_time          2022-10-27T13:32:09.410419432Z
meta                map[role:k8s-gcp-env]
num_uses            0
orphan              true
path                auth/jwt/login
policies            [default k8s-gcp-env]
renewable           true
ttl                 59s
type                service
$ export SERVICE_ACCOUNT="$(vault kv get -field=service_account gitlab/k8s-gcp-env/gcp)"
Error reading gitlab/data/k8s-gcp-env/gcp: Error making API request.
URL: GET https://vault.__REDACTED__/v1/gitlab/data/k8s-gcp-env/gcp
Code: 403. Errors:
* 1 error occurred:
    * permission denied
$ echo $SERVICE_ACCOUNT

So I decided to debug it outside the pipline with an ad-hoc token with the same policy:

❯ vault token create -policy=k8s-gcp-env
Key                  Value
---                  -----
token                __REDACTED__
token_accessor       __REDACTED__
token_duration       768h
token_renewable      true
token_policies       ["default" "k8s-gcp-env"]
identity_policies    []
policies             ["default" "k8s-gcp-env"]

❯ VAULT_TOKEN="__REDACTED__" vault token lookup
Key                 Value
---                 -----
accessor            __REDACTED__
creation_time       1666898416
creation_ttl        768h
display_name        token
entity_id           n/a
expire_time         2022-11-28T19:20:16.462740878Z
explicit_max_ttl    0s
id                  __REDACTED__
issue_time          2022-10-27T19:20:16.462747868Z
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [default k8s-gcp-env]
renewable           true
ttl                 767h58m30s
type                service

vault token capabilities __REDACTED__ gitlab/k8s-gcp-env/gcp
list, read

(however despite having the capabilities, this fails)

VAULT_TOKEN="__REDACTED__" vault kv get -field=service_account gitlab/k8s-gcp-env/gcp
Error reading gitlab/data/k8s-gcp-env/gcp: Error making API request.

URL: GET https://vault.__REDACTED__/v1/gitlab/data/k8s-gcp-env/gcp
Code: 403. Errors:

* 1 error occurred:
    * permission denied

The secret path is definitely correct as when I do this with a root token, it works:

❯ vault kv get -field=service_account gitlab/k8s-gcp-env/gcp
{
  "type": "service_account",
  "project_id": "__REDACTED__",
  "private_key_id": "__REDACTED__",
  "private_key": "__REDACTED__",
  "client_email": "__REDACTED__",
  "client_id": "__REDACTED__",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/__REDACTED__"
}

What am I missing? This is driving me mad.

This is the page I've been following: https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/

Answer 1

Answer was to grant the policy capabilities to gitlab/data/k8s-gcp-env:

vault policy write k8s-gcp-env - <<EOF                                                                                                                                              
path "gitlab/data/k8s-gcp-env/*" {
        capabilities = [ "read", "list" ]
}
EOF

I'm not really sure of the "why" here, but it works.