Is it possible to mount a shared Azure disk in Azure Kubernetes to multiple PODs/Nodes?

Question

I want to mount an Azure Shared Disk to the multiple deployments/nodes based on this: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-shared

So, I created a shared disk in Azure Portal and when trying to mount it to deployments in Kubernetes I got an error:

"Multi-Attach error for volume "azuredisk" Volume is already used by pod(s)..."

Is it possible to use Shared Disk in Kubernetes? If so how? Thanks for tips.

Answer 1

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: manual
provisioner: disk.csi.azure.com
parameters:
  skuname: StandardSSD_LRS
  kind: managed
  maxShares: "3"
  cachingMode: None  
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test
spec:
  capacity:
    storage: 256Gi
  volumeMode: Block
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain 
  storageClassName: manual   
  azureDisk:
    kind: Managed
    diskURI: /subscriptions/b3446H73N3933/resourceGroups/PPFD-RG-STG/providers/Microsoft.Compute/disks/test
    diskName: test 
    cachingMode: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        app: test
      containers: 
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeDevices:
        - name: test
          devicePath: /data
      volumes:
      - name: test
        persistentVolumeClaim:
          claimName: pod-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
      name: pod-claim
spec:
    resources:
        requests:
          storage: 256Gi
    volumeMode: Block
    storageClassName: "manual"
    volumeName: test
    accessModes:
        - ReadWriteMany

we have tried above yamls but we are getting below error.

Warning  FailedAttachVolume  49s (x15 over 15m)  attachdetach-controller  AttachVolume.Attach failed for volume "test" : rpc error: code = InvalidArgument desc = Volume capability not supported.

Answer 2

Note Only raw block device(volumeMode: Block) is supported on shared disk feature, Kubernetes application should manage coordination and control of writes, reads, locks, caches, mounts, fencing on the shared disk which is exposed as raw block device. Multi-node read write is not supported by common file systems (e.g. ext4, xfs), it's only supported by cluster file systems.

details: https://github.com/kubernetes-sigs/azuredisk-csi-driver/tree/master/deploy/example/sharedisk

Answer 3

Yes, you can, and the capability is GA.

An Azure Shared Disk can be mounted as ReadWriteMany, which means you can mount it to multiple nodes and pods. It requires the Azure Disk CSI driver, and the caveat is that currently only Raw Block volumes are supported, thus the application is responsible for managing the control of writes, reads, locks, caches, mounts, and fencing on the shared disk, which is exposed as a raw block device. This means that you mount the raw block device (disk) to a pod container as a volumeDevice rather than a volumeMount.

The documentation examples mostly points to how to create a Storage Class to dynamically provision the static Azure Shared Disk, but I have also created one statically and mounted it to multiple pods on different nodes.

Dynamically Provision Shared Azure Disk

1. Create Storage Class and PVC

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-csi
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_LRS  # Currently shared disk only available with premium SSD
  maxShares: "2"
  cachingMode: None  # ReadOnly cache is not available for premium SSD with maxShares>1
reclaimPolicy: Delete
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azuredisk
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 256Gi  # minimum size of shared disk is 256GB (P15)
  volumeMode: Block
  storageClassName: managed-csi

2. Create a deployment with 2 replicas and specify volumeDevices, devicePath in Spec

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: deployment-azuredisk
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: deployment-azuredisk
    spec:
      containers:
        - name: deployment-azuredisk
          image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
          volumeDevices:
            - name: azuredisk
              devicePath: /dev/sdx
      volumes:
        - name: azuredisk
          persistentVolumeClaim:
            claimName: pvc-azuredisk

Use a Statically Provisioned Azure Shared Disk

Using an Azure Shared Disk that has been provisioned through ARM, Azure Portal, or through the Azure CLI.

1. Define a PersistentVolume (PV) that references the DiskURI and DiskName:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: azuredisk-shared-block
spec:
  capacity:
    storage: "256Gi" # 256 is the minimum size allowed for shared disk
  volumeMode: Block # PV and PVC volumeMode must be 'Block'
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  azureDisk:
    kind: Managed
    diskURI: /subscriptions/<subscription>/resourcegroups/<group>/providers/Microsoft.Compute/disks/<disk-name>
    diskName: <disk-name>
    cachingMode: None # Caching mode must be 'None'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-azuredisk-managed
spec:
  resources:
    requests:
      storage: 256Gi
  volumeMode: Block
  accessModes:
    - ReadWriteMany
  volumeName: azuredisk-shared-block # The name of the PV (above)

Mounting this PVC is the same for both dynamically and statically provisioned shared disks. Reference the deployment above.