Kubernetes Volume Binding results in stale data

Question

My Kubernetes Deployment is composed like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: "{{ .Release.Name }}-{{ .Values.web.service.name }}"
  namespace: "{{ .Values.namespace }}"
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      name: "{{ .Values.web.deployment.selector }}"
  template:
    metadata:
      labels:
        name: "{{ .Values.web.deployment.selector }}"
    spec:
      {{- if $.Values.vault.serviceAccount }}
      serviceAccountName: "{{ $.Release.Name }}-vault-auth"
      automountServiceAccountToken: true
      {{- end }}
      volumes:
        - name: shared-data
          emptyDir: {}
        - name: vault-token
          emptyDir:
            medium: Memory
        - name: company-config
          configMap:
            name: "{{ .Release.Name }}-config"
            items:
              - key: companyRootCA.crt
                path: companyRootCA.crt
        - name: vault-consul-config
          configMap:
            name: "{{ .Release.Name }}-vault-configmap"
            items:
              - key: vault_agent.hcl
                path: vault_agent.hcl
              - key: consul_template_config.hcl
                path: consul_template_config.hcl
              - key: config.tmpl
                path: config.tmpl
      containers:
        - name: vault-agent-auth
          image: vault
          volumeMounts:
            - name: company-config
              mountPath: /etc/pki/ca-trust/source/anchors/companyRootCA.crt
              subPath: companyRootCA.crt
            - name: vault-consul-config
              mountPath: /etc/vault/vault_agent.hcl
              subPath: vault_agent.hcl
            - name: vault-token
              mountPath: /home/vault/
          env:
            - name: VAULT_ADDR
              value: "{{ .Values.vault.endpoint }}"
            - name: VAULT_NAMESPACE
              value: "company/devops/tarchon/{{ .Values.environmentName }}"
          args:
            [
              "agent",
              "-config=/etc/vault/vault_agent.hcl",
              "-log-level=debug"
            ]
        - name: consul-template
          image: hashicorp/consul-template:alpine
          imagePullPolicy: Always
          volumeMounts:
            - name: company-config
              mountPath: /etc/pki/ca-trust/source/anchors/companyRootCA.crt
              subPath: companyRootCA.crt
            - name: vault-consul-config
              mountPath: /etc/consul-template/consul_template_config.hcl
              subPath: consul_template_config.hcl
            - name: vault-token
              mountPath: /home/vault
            - name: vault-consul-config
              mountPath: /etc/templates/config.tmpl
              subPath: config.tmpl
            - name: shared-data
              mountPath: /etc/secrets
          env:
            - name: HOME
              value: /home/vault
            - name: VAULT_ADDR
              value: "{{ .Values.vault.endpoint }}"
            - name: VAULT_NAMESPACE
              value: "company/devops/tarchon/{{ .Values.environmentName }}"
          args:
            [
              "-config=/etc/consul-template/consul_template_config.hcl",
              "-log-level=trace",
            ]
        - name: "{{ .Values.web.service.name }}"
          image: "{{ .Values.image.registry }}:{{ .Values.image.tag }}"
          imagePullPolicy: "{{ .Values.image.imagePullPolicy }}"
          args: [
            "bash", 
            "-c", 
            "python manage.py collectstatic --noinput && gunicorn --bind :8000 --workers 3 ecops_cross_team_platform_backend.wsgi:application"
          ]
          volumeMounts:
            - name: shared-data
              mountPath: /usr/src/app/config.json
              subPath: config.json
          {{- if $.Values.environmentVariables }}
          env:
            {{- range $key, $value := $.Values.environmentVariables }}
            - name: {{ $key }}
              valueFrom:
                configMapKeyRef:
                  name: "{{ $.Release.Name }}-config"
                  key: {{ $key | quote }}
            {{- end }}
          {{- end }}
          ports:
            - containerPort: {{ .Values.web.service.port }}
          resources: {}
      restartPolicy: Always
status: {}

consul-template generates a file /etc/secrets/config.json with credentials that come from HashiCorp Vault in the shared volume shared-data.

In my application container I bind the file in a different directory (/usr/src/app/config.json) (because the application wants the file to be in different directory compared to where consul-template generates the file).

The problem is that whenever the file config.json is updated in the volume that is mounted in the consul-template container, the modification is not propagated to the other containers, so I end up having the application container with stale non-working data.

At the beginning, I initially thought it was a problem caused by the readOnly volumeMount option, however the problem was still present after removing it.

Answer 1

As anmol said in the comments, it was indeed the fact of binding a single file using subPath in the directory.

The solution was to remove the subPath and bind the shared-data volume into a single-scoped folder (ie. /usr/src/app/credentials) so that it won't go wrong with other things.

Solution:

volumeMounts:
  - name: shared-data
    mountPath: /usr/src/app/credentials

Instead of:

volumeMounts:
  - name: shared-data
    mountPath: /usr/src/app/config.json
    subPath: config.json