CODEWITHSUNDEEP
azurekuberneteskubernetes-helmazure-aksazure-keyvault
X T
X T

Reputation: 486

key vault issue with AKS

A couple of weeks ago i published similar question regarding a Kubernetes deployment that uses Key Vault (with User Assigned Managed identity method). The issue was resolved but when trying to implemente everything from scratch something makes not sense to me.

Basically i am getting this error regarding mounting volume:

Volumes:
  sonar-data-new:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-data-new
    ReadOnly:   false
  sonar-extensions-new2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-extensions-new2
    ReadOnly:   false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-provider
  default-token-zwxzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwxzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned default/sonarqube-d44d498f8-46mpz to aks-agentpool-35716862-vmss000000
  Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-d44d498f8-46mpz, err: rpc error: code = Unknown desc = failed to mountobjects, error: failed to get objectType:secret, objectName:username, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://SonarQubeHelm.vault.azure.net/secrets/username/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}

This is my secret-class.yml file (name of the keyvault is correct). Also xxx-xxxx-xxx-xxx-xxxxx4b5ec83 is the objectID of the AKS managed identity (SonarQubeHelm-agentpool)

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxx-xxxx-xxx-xxx-xxxxx4b5ec83"
    keyvaultName: "SonarQubeHelm"
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: username
          objectType: secret
          objectAlias: username
          objectVersion: ""
        - |
          objectName: password
          objectType: secret
          objectAlias: password
          objectVersion: ""
    resourceGroup: "rg-LD-sandbox"
    subscriptionId: "xxxx"
    tenantId: "yyyy"

and this is my deployment.yml file

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

I assigned proper permissions to the AKS managed identity to get access to the keyvault (xxx-xxxx-xxx-xxx-xxxxx4b5ec83 is the objectID of the AKS managed identity - SonarQubeHelm-agentpool)

xxxx@Azure:~/clouddrive/kubernetes/sonarqubekeyvault$ az role assignment list --assignee xxx-xxxx-xxx-xxx-xxxxx4b5ec83 --all
[
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxx-xxx-xxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-x-xx-xx-xx86584218f",
    "name": "xxxx-xx-x-x-xx86584218f",
    "principalId": "xxx-xxx-x-xxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxx-xxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxx-xxxx-xxx-xxxx-xxxfe8e74483",
    "roleDefinitionName": "Key Vault Administrator",
    "scope": "/subscriptions/xxxx-xxx-xxx-xxxx-xxx2e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  },
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-xxxx-xxxx-xxxx-xxx5137f480",
    "name": "xxxx-xxxx-xxxx-xxxx-xx5137f480",
    "principalId": "xxxx-xxxx-xxxx-xxxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxxx-xxxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxxx-xxxx-xxxx-xxxx-xx2c155cd7",
    "roleDefinitionName": "Key Vault Secrets Officer",
    "scope": "/subscriptions/xxxx/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  }
]

This is the info about my Key Vault.

az keyvault show --name SonarQubeHelm

{
  "id": "/subscriptions/xxxx-xxxxx-xxxx-xxxx-xxxxxxxx804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
  "location": "xxxxx",
  "name": "SonarQubeHelm",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "xxxx-xxx-xxxx-xxxx-xxxxa4b5ec83",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxx-xxxx-xxxx-xxxx-xxxxxdb8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxxx-xxxx-xxxx-xxxx-xxxx531f67f8",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxxx-xxxxxx-xxx-xxx8db8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxx-xxxx-xxxx-xxx-xxxx0df6af9",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "storage": null
        },
        "tenantId": "xxx-xxx-xxx-xxx-xxx8db8c610"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": false,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": false,
    "enabledForTemplateDeployment": false,
    "hsmPoolResourceId": null,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",
    "sku": {
      "family": "A",
      "name": "Standard"
    },
    "softDeleteRetentionInDays": 90,
    "tenantId": "xxx-xxx-xxx-xxx-xxx68db8c610",
    "vaultUri": "https://sonarqubehelm.vault.azure.net/"
  },
  "resourceGroup": "rg-LD-sandbox",
  "systemData": null,
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

This is the CSI pod running at the moment:

NAME                                                READY   STATUS              RESTARTS   AGE
csi-secrets-store-provider-azure-1632148185-hggl4   1/1     Running             0          5h4m
ingress-nginx-controller-65c4f84996-99pkh           1/1     Running             0          5h49m
secrets-store-csi-driver-xsx2r                      3/3     Running             0          5h4m
sonarqube-d44d498f8-46mpz                           0/1     ContainerCreating   0          26m

I used this CSI driver

helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --set secrets-store-csi-driver.syncSecret.enabled=true --generate-name

To assign proper permissions to the AKS managed identity i followed (just in case i used clientID as well, but it does not work). However, from the previous commands permissions to managed identity seems correct.

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

In Access Policies in the key vault "SonarQubeHelm" these are the applications added with all the permission: SonarQubeHelm-agentpool and SonarQubeHelm

The key values in Secrets (inside Key Vault) are username and password.

Everything is on the same region, resource group and namespace (default) and I am working with 1 node cluster.

Any idea about this error?

Thanks in advance!

Upvotes: 1

Views: 2810

Answers (1)

X T
X T

Reputation: 486

After doing some tests, it seems that the process that I was following was correct. Most probably, I was using principalId instead of clientId in role assignment for the AKS managed identity.

Key points for someone else that is facing similar issues:

  1. Check what the managed identity created automatically by AKS is. Check for the clientId; e.g.,

    az vmss identity show -g MC_rg-LX-sandbox_SonarQubeHelm_southcentralus -n 
aks-agentpool-xxxxx62-vmss -o yaml

    Remember that MC_** is the resource group that AKS creates automatically to keep AKS resources. VMSS is the Virtual Machine Scale Set; you can get it under the same resource group.

  2. Check if it has correct permissions to access the Key Vault that you created: e.g., (where xxxx-xxxx-xxx-xxx-xx79a3906b8 is the managed identity clientId):

    az role assignment list --assignee xxxx-xxxx-xxx-xxx-xx79a3906b8 --all

    It should have:

    "roleDefinitionName": "Key Vault Administrator"

  3. If it doesn't have correct permissions, assign them:

    export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.clientId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

The role assignments for the managed identity in AKS works for clientId.

Upvotes: 3

Related Questions