CODEWITHSUNDEEP
kubernetesvault
Meet101
Meet101

Reputation: 801

Enable k8s pods from multiple namespaces to read vault secret through vault agent

I have a vault setup in k8s with k8s auth enabled to allow vault agent to read secrets and export them as an environment variables to a k8s pod using K8s service account. everything is working fine if I’m using a single k8s namespace.

I am not able to use a service account from A namespace and trying to use it in B namespace after attaching it via a rolebinding in namespace B

step 1 - I created a service account called vault-ro in default namespace and configured it in vault k8s auth role. everything works good for any k8s pod in default namespace. they are able to read secerts from vault.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-ro

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
   name: role-tokenreview-binding ##This Role!
   namespace: default
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-ro
  namespace: default

now, I want to enable namespace B to use same vault role and k8s service account to read secret from vault. so i created a rolebinding as follow in namespace B

role binding in Namespace B

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
   name: role-tokenreview-binding-dev
   namespace: B
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-ro
  namespace: default

expected behaviour, I should be able to spin up a k8s pod with vault-ro service account user and it should be able to read the secret from vault same way as it does in default namespace but when I try that, i’m getting error as

Error from server (Forbidden): error when creating "test-app-nonprod.yaml": pods "test-app" is forbidden: error looking up service account B/vault-ro: serviceaccount "vault-ro" not found

why it’s not able to reference service account vault-ro from default namespace and still trying to find if it’s present in B namespace? is it something to do with vault? I tried my best to find from everywhere, all documents saying above should work!

appreciate any help!

Upvotes: 2

Views: 1388

Answers (1)

trmatthe
trmatthe

Reputation: 66

Your error message is saying that your pod can't find service account vault-ro in the B namespace.

error looking up service account B/vault-ro

Are you setting a pod.spec.serviceAccountName entry in your yaml? If so, the service account must exist in the same namespace as the pod is running. Both Pods and ServiceAccounts are namespaced.

I can't give you a good link to the docs where this is stated (that the sa must exist, it's implied in a few places. To check whether an object is namespaced or not you can use https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#not-all-objects-are-in-a-namespace) but I learnt this through experience.

I would create another service account in namespace B and do the rolebinding again.

As an aside, your have a mix of versions in your yaml for rbac, so if you want to avoid having this possibly bite you in the future if v1beta gets deprecated, it's worth tidying that too. Also, you are doing a ClusterRoleBinding in the first half, and a RoleBinding in the second which isn't necessary.

I'd use this manifest and change it for each serviceaccount as indicated:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
   name: role-tokenreview-binding 
   namespace: default       #*** change 1
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-ro
  namespace: default        #*** change 2

Create a serviceaccount in each namespace you need to deploy vault-accessing pods in, and update the namespaces marked for each namespace you create an sa in.

Upvotes: 2

Related Questions