Deploy bitnami redis-cluster using helm

Question

I have done bitnami redis-cluster deployment using helm chart in kubernetes. https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster

But it gives following permission issue after pod running some time.

1:M 12 Feb 2022 16:49:58.886 * Starting automatic rewriting of AOF on 157422% growth
1:M 12 Feb 2022 16:49:58.887 * Background append only file rewriting started by pid 6625
6625:C 12 Feb 2022 16:49:58.887 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied
1:M 12 Feb 2022 16:49:58.987 # Background AOF rewrite terminated with error
1:M 12 Feb 2022 16:49:59.088 * Starting automatic rewriting of AOF on 157422% growth
1:M 12 Feb 2022 16:49:59.089 * Background append only file rewriting started by pid 6626
6626:C 12 Feb 2022 16:49:59.089 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied

here is the statefulset template which is generated by helm to master node.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-master
  namespace: "redis"
  labels:
    app.kubernetes.io/name: redis
    helm.sh/chart: redis-15.6.3
    app.kubernetes.io/instance: redis
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: master
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: redis
      app.kubernetes.io/instance: redis
      app.kubernetes.io/component: master
  serviceName: redis-headless
  updateStrategy:
    rollingUpdate: {}
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: redis
        helm.sh/chart: redis-15.6.3
        app.kubernetes.io/instance: redis
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: master
      annotations:
        checksum/configmap:
        checksum/health: 
        checksum/scripts:
        checksum/secret: 
    spec:
      securityContext:
        fsGroup: 1001
      serviceAccountName: redis
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: redis
                    app.kubernetes.io/instance: redis
                    app.kubernetes.io/component: master
                namespaces:
                  - "redis"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      terminationGracePeriodSeconds: 30
      containers:
        - name: redis
          image: docker.io/bitnami/redis:6.2.6-debian-10-r53
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsUser: 1001
          command:
            - /bin/bash
          args:
            - -c
            - /opt/bitnami/scripts/start-scripts/start-master.sh
          env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: REDIS_REPLICATION_MODE
              value: master
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
            - name: REDIS_TLS_ENABLED
              value: "no"
            - name: REDIS_PORT
              value: "6379"
          ports:
            - name: redis
              containerPort: 6379
          livenessProbe:
            initialDelaySeconds: 20
            periodSeconds: 5
            # One second longer than command timeout should prevent generation of zombie processes.
            timeoutSeconds: 6
            successThreshold: 1
            failureThreshold: 5
            exec:
              command:
                - sh
                - -c
                - /health/ping_liveness_local.sh 5
          readinessProbe:
            initialDelaySeconds: 20
            periodSeconds: 5
            timeoutSeconds: 2
            successThreshold: 1
            failureThreshold: 5
            exec:
              command:
                - sh
                - -c
                - /health/ping_readiness_local.sh 1
          resources:
            limits: {}
            requests: {}
          volumeMounts:
            - name: start-scripts
              mountPath: /opt/bitnami/scripts/start-scripts
            - name: health
              mountPath: /health
            - name: redis-data
              mountPath: /data
              subPath: 
            - name: config
              mountPath: /opt/bitnami/redis/mounted-etc
            - name: redis-tmp-conf
              mountPath: /opt/bitnami/redis/etc/
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: start-scripts
          configMap:
            name: redis-scripts
            defaultMode: 0755
        - name: health
          configMap:
            name: redis-health
            defaultMode: 0755
        - name: config
          configMap:
            name: redis-configuration
        - name: redis-tmp-conf
          emptyDir: {}
        - name: tmp
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: redis-data
        labels:
          app.kubernetes.io/name: redis
          app.kubernetes.io/instance: redis
          app.kubernetes.io/component: master
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "8Gi"

Apparently they have given permission to data folder.

Answer 1

UPDATED-2: I found a clue that this issue is related with the config dir get result, but don't know how it become to this.

normal: redis-cli config get dir

1) "dir"
2) "/bitnami/redis/data"

abnormal: redis-cli config get dir

1) "dir"
2) "/etc"

Because the dir has been changed from /bitnami/redis/data to /etc, it's reasonable that the permission was denied.

Moreover, When it happened, the SYNC command between master & slave might be failed by this log, too: (master's log) Failed opening the RDB file crontab (in server root dir /etc) for saving: Permission denied

Use this command will fix the permission problem:

redis-cli config set dir /bitnami/redis/data

And then everything works again...

I haven't found the root cause yet. But I have opened an issue for this.

UPDATED: Sorry, my solution didn't work. After running a few days, this problem came out again.

I was struggling on this problem this morning, too. And then I found someone mentioned the setting VolumePermissions in this issue's comment.

I did miss this config from my chart. So I quickly compare the statefulset yaml between enabling VolumePermissions or not, and then found the following diffs:

      enableServiceLinks: false
# ------------------ diff starts from here ------------------
      initContainers:
      - command:
        - /bin/chown
        - -R
        - 1001:1001
        - /bitnami/redis/data
        image: docker.io/bitnami/minideb:buster
        imagePullPolicy: Always
        name: volume-permissions
        resources: {}
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami/redis/data
          name: redis-data
# ----------------------------------------------
      restartPolicy: Always

I copied the diff blocks to my old statefulset, and the redis-cluster's pods began to restart one by one.

So far, the problem is solved. I'm not sure what I did is right, but at least this made my BGREWRITEAOF cmd work again.