Where is Kubernetes storage location of a Persistent Volume on Docker Desktop for mac?

Question

I'm not sure if I have a configuration error or I just don't know where to look but I can't seem to find where my files are being stored with these configurations

apiVersion: v1
kind: PersistentVolume
metadata:
  name: tmp-storage
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /tmp/netes
    type: Directory

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tmp-storage-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

Deployment

spec:
  volumes:
    - name: abcd
      persistentVolumeClaim:
       claimName: tmp-storage-claim 

  containers:
    ...
    volumeMounts:
      - mountPath: '/tmp'
        name: abcd

I've tried accessing the docker desktop vm with docker run -it --rm --privileged --pid=host justincormack/nsenter1 but there is nothing inside /tmp

There's also nothing on my computer inside /tmp

I've also tried looking in /Users/username/.docker/Volumes/ but I have no volumes directory in that location

Answer 1

If you're using Docker Desktop with Kubernetes, the Persistant Volume will be on the Node, and not the local machine. The easiest way to get into that Node and view the filesystem is to run the below docker command. Once that is done, you can then use the terminal to explore a little bit.

I am running Kubernetes 1.22.5 and Docker Desktop 4.5.0

docker run -it --rm --privileged --pid=host justincormack/nsenter1

Here is an example of a Persistent Volume that I have configured. Note that my hostPath is "mnt/data"

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  namespace: my-namespace
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

The confusing part is where to start looking for the PV. The easiest way to do that is to just use find in your terminal to search for that directory.

In my example, I searched for the "data" directory:

find / -type d -name "data"

This will obviously return a few results. You will have to take a look at the result set and find the one that matches your path, but in my case, I was the last result. My PV was located at

/containers/services/docker/rootfs/mnt/data

Note that the hostPath only works on Single Node Clusters.

Answer 2

Here how I found it on my Mac:

1. Create PV, PVC, Deployment as you've mentioned. I've just change the PV spec.hostPath.type to DirectoryOrCreate

2. Create a file on the volume using pod shell:

kubeclt exec -ti the-deployment-pod-name-here -- touch /tmp/somefile2.txt

3. Run nsenter pod:

docker run -it --rm --privileged --pid=host alpine:edge nsenter -t 1 -m -u -n -i sh

(on the recent DockerDesktop 3.5.1
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
doesn't show me a VM shell prompt any more)

4. Find the file using find in the nsenter container filesystem:

/ # find / -name somefile2.txt
/var/lib/mount-docker-cache/entries/docker.tar/a08ee16361132af943875aee79aae165cdbfaf0d203ee97fe9d61d29e307aceb/containers/services/docker/tmp/upper/tmp/netes/somefile2.txt
/var/lib/mount-docker-cache/entries/docker.tar/a08ee16361132af943875aee79aae165cdbfaf0d203ee97fe9d61d29e307aceb/containers/services/docker/rootfs/tmp/netes/somefile2.txt
/var/lib/mount-docker-cache/entries/services.tar/bbec751ae945082378b0b2d4a7c6e32f9c35164315587914a95acc4bee8df1ff/containers/services/docker/tmp/upper/tmp/netes/somefile2.txt
/var/lib/mount-docker-cache/entries/services.tar/bbec751ae945082378b0b2d4a7c6e32f9c35164315587914a95acc4bee8df1ff/containers/services/docker/rootfs/tmp/netes/somefile2.txt
/containers/services/docker/tmp/upper/tmp/netes/somefile2.txt
/containers/services/docker/rootfs/tmp/netes/somefile2.txt

5. Most promising paths that should work for most cases are:

/containers/services/docker/tmp/upper
/containers/services/docker/rootfs

+ PV hostPath: /tmp/netes
+ filename: somefile2.txt

Note: HostPath PV files are located in DockerVM filesystem. I haven't found a way to share Mac folder to PV in DockerDesktop Kubernetes Pod for now.

Note2: Without specifying StorageClass in PV and PVC, DockerDesktop Kubernetes assigns a default storage class which in my case was hostpath and generates a folder for the PV in the DockerVM temporary location:

/var/lib/k8s-pvs/tmp-storage-claim/pvc-dafbcdf6-a3e8-47cc-af1a-cf740d8ffcd0/somefile2.txt

Answer 3

Based on this article the solution should be as follow:

docker for Mac runs a virtual machine behind the scenes and hides it from you to make things
simpler. Simpler, unless you want to dig deeper. If you want to access persistent volumes created by Docker you need to login into virtual machine first.

1. We need to "screen into" the Docker driver by executing a command: screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
2. You should see a blank screen, just press Enter , and after a while, you should see a command line prompt:
   docker-desktop:~#
3. Now you're inside Docker's VM and you can cd into volumes dir by typing: cd /var/lib/docker/volumes
4. Profit, you got there!
5. If you need to transfer files from your MacOS host into Docker host (for example to put files into docker volumes) use directories shared between host (mac os) and Docker host (Docker VM), you can find a list of such directories under File Sharing tab of your Docker for Mac application.

Note: file sharing is used to share files one way - from host (Mac) to container. You won't see all files stored in persistent volume this way. You will only see the files that you have specially shared with the container. To see the entire persistent volume you need to follow the steps above.

See also:

• Similar problem on StackOverflow,
• Where is /var/lib/docker on Mac/OS X,
• Docker Desktop for Mac user manual.