WSL Kubernetes Pod stuck in ContainerCreating state due to volume mount

Question

I am working with Docker Desktop on Windows 10 as well as Windows Subsystem for Linux (WSL). I have a containerized app that I deploy to the local K8s cluster (courtesy of Docker Desktop). Typical story: all was working fine until one day a Docker Desktop update came and ruined everything). DD version I have now is 2.3.0.2 stable.

I have a pod with MySQL with defined pv, pvc and storage class. When I deploy my app to the cluster I can see that pv and pvc are bound but the pod is stuck at ContainerCreating state:

$ kubectl describe pod mysql-6779d8fb8b-d25wz
Name:           mysql-6779d8fb8b-d25wz
Namespace:      typo3-connector
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Wed, 13 May 2020 14:21:43 +0200
Labels:         app=mysql
                pod-template-hash=6779d8fb8b
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/mysql-6779d8fb8b
Containers:
  mysql:
    Container ID:
    Image:          lw-mysql
    Image ID:
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      mysql-credentials  Secret  Optional: false
    Environment:         <none>
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wr6g9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-wr6g9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wr6g9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                From                     Message
  ----     ------            ----               ----                     -------
  Warning  FailedScheduling  <unknown>          default-scheduler        persistentvolumeclaim "mysql-pv-claim" not found
  Warning  FailedScheduling  <unknown>          default-scheduler        persistentvolumeclaim "mysql-pv-claim" not found
  Normal   Scheduled         <unknown>          default-scheduler        Successfully assigned typo3-connector/mysql-6779d8fb8b-d25wz to docker-desktop
  Warning  FailedMount       35s (x8 over 99s)  kubelet, docker-desktop  MountVolume.NewMounter initialization failed for volume "mysql-pv" : path "/c/kubernetes/typo3-8/mysql-storage/" does not exist

The error is

MountVolume.NewMounter initialization failed for volume "mysql-pv" : path "/c/kubernetes/typo3-8/mysql-storage/" does not exist

But the path actually exists on disk (in WSL that is).

The pv:

$ kubectl describe pv mysql-pv
Name:              mysql-pv
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"mysql-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capa...
                   pv.kubernetes.io/bound-by-controller: yes
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-storage
Status:            Bound
Claim:             typo3-connector/mysql-pv-claim
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          2Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [docker-desktop]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /c/kubernetes/typo3-8/mysql-storage/
Events:    <none>

The pvc:

$ kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     typo3-connector
StorageClass:  local-storage
Status:        Bound
Volume:        mysql-pv
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"typo3-connector"},"spe...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    mysql-6779d8fb8b-d25wz
Events:        <none>

I tried running it from PowerShell but no luck. I get the same result. But it was all working fine before the update.

Is it configuration based issue?

EDIT Here's the manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: typo3-connector
---
apiVersion: v1
data:
  MYSQL_PASSWORD: ZHVtbXk=
  MYSQL_ROOT_PASSWORD: ZHVtbXk=
  MYSQL_USER: ZHVtbXk=
kind: Secret
metadata:
  name: mysql-credentials
  namespace: typo3-connector
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mysql
  name: mysql
  namespace: typo3-connector
spec:
  ports:
  - name: mysql-backend
    port: 3306
    protocol: TCP
  selector:
    app: mysql
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: typo3-connector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - envFrom:
        - secretRef:
            name: mysql-credentials
        image: mysql:5.6
        imagePullPolicy: IfNotPresent
        name: mysql
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mysql-persistent-storage
      imagePullSecrets:
      - name: lwdockerregistry
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  local:
    path: /c/kubernetes/typo3-8/mysql-storage/
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - docker-desktop
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  namespace: typo3-connector
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: local-storage

Answer 1

You must use this format for the path:

/run/desktop/mnt/host/c/someDir/volumeDir

So in your case it is:

/run/desktop/mnt/host/c/kubernetes/typo3-8/mysql-storage/

source for solution

UPDATE: You really don't want to use windows host directories inside WSL2 distro because it is really really poor performance. Use the WSL2 distro directories instead that you can access just as easily from the host windows on this address \\wsl$

Answer 2

As described in the following example taken from here, the path shouldn't end with /.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
spec:
  capacity:
    storage: 500Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/disks/vol1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node

Changing your path to path: /c/kubernetes/typo3-8/mysql-storage makes the magic and your PVC works as designed.