CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-engine
jcoop7777
jcoop7777

Reputation: 73

GKE Persistent Volume Not Saving Data

I have created a Persistent Volume and Volume claim for an app I am working on in GKE. The claim and storage appear to be setup correctly, however, the data doesn't persist if the pod is restarted. I am able to save data initially and I can see the file in the pod, but it disappears after restarting it.

I had asked this question previously, but didn't include my .yaml files and received a sort of generic answer as a result so I decided to repost with the .yaml files hoping someone could look at them and tell me where I am going wrong. From everything I've seen, it looks like the problem is in the Persistent Volume as the claim looks exactly like everyone else's.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prod-api-meta-uploads-k8s
  namespace: default
  resourceVersion: "4500192"
  selfLink: /apis/apps/v1/namespaces/default/deployments/prod-api-meta-uploads-k8s
  uid: *******
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: prod-api-meta-uploads-k8s
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        gcb-build-id: *****
        gcb-trigger-id:****
      creationTimestamp: null
      labels:
        app: prod-api-meta-uploads-k8s
        app.kubernetes.io/managed-by: gcp-cloud-build-deploy
        app.kubernetes.io/name: prod-api-meta-uploads-k8s
        app.kubernetes.io/version: becdb864864f25d2dcde2e62a2f70501cfd09f19
    spec:
      containers:
      - image: bitbucket.org/api-meta-uploads-k8s@sha256:7766413c0d
        imagePullPolicy: IfNotPresent
        name: prod-api-meta-uploads-k8s-sha256-1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /uploads/profileImages
          name: uploads-volume-prod
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: uploads-volume-prod
        persistentVolumeClaim:
          claimName: my-disk-claim-1
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-09-08T21:00:40Z"
    lastUpdateTime: "2020-09-10T04:54:27Z"
    message: ReplicaSet "prod-api-meta-uploads-k8s-5c8f66f886" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-09-10T06:49:41Z"
    lastUpdateTime: "2020-09-10T06:49:41Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 36
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

** Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2020-09-09T16:12:51Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: uploads-volume-prod
  namespace: default
  resourceVersion: "4157429"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/uploads-volume-prod
  uid: f93e6134
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-f93e6
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  phase: Bound

*** PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  name: my-disk-claim-1
  namespace: default
  resourceVersion: "4452471"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/my-disk-claim-1
  uid: d533702b
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast
  volumeMode: Filesystem
  volumeName: pvc-d533702b
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

Upvotes: 1

Views: 1348

Answers (1)

PjoterS
PjoterS

Reputation: 14084

As you are using GKE you don't need to prepare PersistentVolume and PersistentVolumeClaim manually (static provisioning) in relationship 1:1, as GKE can use Dynamic Volume Provisioning. It's good described in Persistent Volumes

When none of the static PVs the administrator created match a user's PersistentVolumeClaim, the cluster may try to dynamically provision a volume specially for the PVC.

In GKE on the beginning you have at least one storageclass named standard. It also have (default) next to name.

$ kubectl get sc
NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   110m

It means that if you won't specify storageClassName in your PersistentVolumeClaim, it will use storageclass which is set as default. In your YAMLs I can see that you have used storageClassName: standard. If you will check this storageclass you will se what ReclaimPolicy was set to delete. Below output:

$ kubectl describe sc standard
Name:                  standard
IsDefaultClass:        Yes
Annotations:           storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/gce-pd
Parameters:            type=pd-standard
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

IsDefaultClass: means this storageclass is set as default.

ReclaimPolicy: defining ReclaimPolicy, delete in this case

As ReclaimPolicy is set to Delete:

For volume plugins that support the Delete reclaim policy, deletion removes both the PersistentVolume object from Kubernetes, as well as the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume. Volumes that were dynamically provisioned inherit the reclaim policy of their StorageClass, which defaults to Delete. The administrator should configure the StorageClass according to users' expectations; otherwise, the PV must be edited or patched after it is created.

Depends on your needs, you can use:

Recycle:

If supported by the underlying volume plugin, the Recycle reclaim policy performs a basic scrub (rm -rf /thevolume/*) on the volume and makes it available again for a new claim. However, please keep in mind that: Warning: The Recycle reclaim policy is deprecated. Instead, the recommended approach is to use dynamic provisioning. Adding this option as I didnt see what K8s version are you using, however it's not supported on GKE.

Retain:

The Retain reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume.

In addition, as you are using GKE, it supports only Delete and Retain.

The StorageClass "another-storageclass" is invalid: reclaimPolicy: Unsupported value: "Recycle": supported values: "Delete", "Retain"

In addition, as you specfied revisionHistoryLimit: 10, pod after 10 restarts will be recreated, in that situation pod, pv and pvc will be deleted when ReclaimPolicy will be set as delete.

Solution

As easiest solution, you should create new StorageClass with ReclaimPolicy different from Delete and use it in your PVC.

$ kubectl get sc,pv,pvc -A
NAME                                               PROVISIONER            AGE
storageclass.storage.k8s.io/another-storageclass   kubernetes.io/gce-pd   53s
storageclass.storage.k8s.io/standard (default)     kubernetes.io/gce-pd   130m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS           REASON   AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            Retain           Bound    tst-dev/pvc-1     another-storageclass            43s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            Retain           Bound    tst-stage/pvc-2   another-storageclass            42s

NAMESPACE   NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
tst-dev     persistentvolumeclaim/pvc-1   Bound    pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            another-storageclass   46s
tst-stage   persistentvolumeclaim/pvc-2   Bound    pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            another-storageclass   45s

$ kubectl delete pvc pvc-1 -n tst-dev
persistentvolumeclaim "pvc-1" deleted
user@cloudshell:~ (project)$ kubectl delete pvc pvc-2  -n tst-stage
persistentvolumeclaim "pvc-2" deleted
user@cloudshell:~ (project)$ kubectl get sc,pv,pvc -A
NAME                                               PROVISIONER            AGE
storageclass.storage.k8s.io/another-storageclass   kubernetes.io/gce-pd   7m49s
storageclass.storage.k8s.io/standard (default)     kubernetes.io/gce-pd   137m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM             STORAGECLASS           REASON   AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            Retain           Released   tst-dev/pvc-1     another-storageclass            7m38s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            Retain           Released   tst-stage/pvc-2   another-storageclass            7m37s

Upvotes: 1

Related Questions