CODEWITHSUNDEEP
kuberneteskubernetes-podpersistent-volumespersistent-volume-claims
camelia
camelia

Reputation: 41

Kubernetes Persistent Volume: MountPath directory created but empty

I have 2 pods, one that is writing files to a persistent volume and the other one supposedly reads those files to make some calculations.

The first pod writes the files successfully and when I display the content of the persistent volume using print(os.listdir(persistent_volume_path)) I get all the expected files. However, the same command on the second pod shows an empty directory. (The mountPath directory /data is created but empty.)

This is the TFJob yaml file:

apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: pod1
  namespace: my-namespace
spec:
  cleanPodPolicy: None
  tfReplicaSpecs:
    Worker:
      replicas: 1
      restartPolicy: Never
      template:
        spec:
          containers:
            - name: tensorflow
              image: my-image:latest
              imagePullPolicy: Always
              command:
                - "python"
                - "./program1.py"
                - "--data_path=./dataset.csv"
                - "--persistent_volume_path=/data"
              volumeMounts:
                - mountPath: "/data"
                  name: my-pv
          volumes:
            - name: my-pv
              persistentVolumeClaim:
                claimName: my-pvc

(respectively pod2 and program2.py for the second pod)

And this is the volume configuration:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  namespace: my-namespace
  labels:
    type: local
    app: tfjob
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
  namespace: my-namespace
  labels:
    type: local
    app: tfjob
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data"

Does anyone have any idea where's the problem exactly and how to fix it?

Upvotes: 2

Views: 3288

Answers (1)

Jonas
Jonas

Reputation: 129075

When two pods should access a shared Persistent Volume with access mode ReadWriteOnce, concurrently - then the two pods must be running on the same node since the volume can only be mounted on a single node at a time with this access mode.

To achieve this, some form of Pod Affinity must be applied, such that they are scheduled to the same node.

Upvotes: 2

Related Questions