CODEWITHSUNDEEP
kuberneteskubernetes-helmazure-akscsi
Vowneee
Vowneee

Reputation: 1459

Helm upgrade is making deployment failure

We configured CSI driver in our cluster for secret management and used the below secret provider class template to automatically assign secrets to the deployments env variable. The above setup is working fine.

But 2 things where I have issues. Whenever new changes were done to the secret, say if adding a new secret to the YAML and key vault, the next release will fail with the helm upgrade command, stating specified secret is not found.

So in order to solve this, I have to uninstall all helm releases and need to install the helm release again, which means down time, how can I achieve this scenario without any down time?

Secondly, is there any recommended way to restart the Pods when the secret template changes:

values.yaml for MyAppA

keyvault:
  name: mykv
  tenantId: ${tenantId}$
  clientid: "#{spid}#"
  clientsecret: "#{spsecret}#"
  secrets:
    - MyAPPA_SECRET1_NAME1
    - MyAPPA_SECRET2_NAME2
    - MyAPPA_SECRET3_NAME3

deployment.yaml, ENV part is as below

    {{- if eq .Values.keyvault.enabled true }}
    {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
      - name: {{ . }}
        valueFrom:
          secretKeyRef:
            name: {{ $.Release.Name }}-kvsecret
            key: {{ . }}
      {{- end }}
      {{- end }}
      volumeMounts: 
      - name:  {{ $.Release.Name }}-volume
        mountPath: '/mnt/secrets-store'
        readOnly: true
  volumes:
    - name: {{ $.Release.Name }}-volume
      csi:
        driver: 'secrets-store.csi.k8s.io'
        readOnly: true
        volumeAttributes:
          secretProviderClass: {{ $.Release.Name }}-secretproviderclass
        nodePublishSecretRef:
          name: {{ $.Release.Name }}-secrets-store-creds

secretProviderClass yaml file is as below.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: {{ $.Release.Name }}-secretproviderclass
  labels:
    app: {{ $.Release.Name }}
    chart: "{{ $.Release.Name }}-{{ .Chart.Version }}"
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  provider: azure
  secretObjects:
  - data:
      {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}
    - key: {{ . }}
      objectName: {{ $.Release.Name | upper }}-{{ . }} 
      {{- end }}      
    secretName: {{ $.Release.Name }}-kvsecret
    type: opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: {{ .Values.keyvault.name | default "mydev-kv" }}   
    objects: |
      array:
        {{- range .Values.keyvault.secrets }}{{/* <-- only one range loop */}}        
        - |
          objectName: {{ $.Release.Name | upper }}-{{ . }}
          objectType: secret
        {{- end }}
    tenantId: {{ .Values.keyvault.tenantid }}
{{- end }}
{{- end -}}
{{- define "commonobject.secretproviderclass" -}}
{{- template "commonobject.util.merge" (append . "commonobject.secretproviderclass.tpl") -}}
{{- end -}}

Upvotes: 0

Views: 1357

Answers (1)

damian5996
damian5996

Reputation: 51

The problem is not in the "helm upgrade" command. I discovered this is a limitation of a CSI driver or SecretProviderClass. When the deployment is already created, the SecretProviderClass resource is updated but the "SecretProviderClassPodStatuses" is not, so secrets are not updated.

Two potential solutions to update secrets:

  • delete secret and restart/recreate pod => this works but it sounds more like a workaround than an actual solution
  • set enableSecretRotation to true => it has been implemented in a CSI driver recently and it's in an 'alpha' version

https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html

Edited:

In the end, I used this command to use automatic secret rotation in Azure Kubernetes Service:

az aks addon update -g [resource-group] -n [aks-name] -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 0.5m

You can use the following command to check if this option is enabled:

az aks addon show -g [resource-group] -n [aks-name] -a azure-keyvault-secrets-provider

More info here: https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#enable-and-disable-autorotation

Upvotes: 2

Related Questions