How can I grant permission of persisted volume to ES container mount path?

Question

I deploy Elasticsearch container to kubernete with a persisted storage. Below is the configuration. After deploy I got this error. How can I grant permission to ES container about the storage?

ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
    at java.base/java.nio.file.Files.createDirectory(Files.java:694)
    at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:801)
    at java.base/java.nio.file.Files.createDirectories(Files.java:787)
    at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275)
    at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212)
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272)
    at org.elasticsearch.node.Node.<init>(Node.java:362)
    at org.elasticsearch.node.Node.<init>(Node.java:289)
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/my-elastic-cluster.log

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 512Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-bd0e5b85 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 512Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: es-config
data:
  elasticsearch.yml: |
    cluster.name: my-elastic-cluster
    network.host: "0.0.0.0"
    bootstrap.memory_lock: false
    discovery.zen.ping.unicast.hosts: elasticsearch-cluster
    discovery.zen.minimum_master_nodes: 1
    discovery.type: single-node
    xpack.security.enabled: true
    xpack.monitoring.enabled: true
    xpack.security.authc.realms:
        native.realm1:
            order: 0
            cache.ttl: 10m 
  ES_JAVA_OPTS: -Xms2g -Xmx4g
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: default
spec:
  serviceName: es-entrypoint
  replicas: 1
  selector:
    matchLabels:
      name: es
  template:
    metadata:
      labels:
        name: es
    spec:
      volumes:
        - name: es-config
          configMap:
            name: es-config
            items:
              - key: elasticsearch.yml
                path: elasticsearch.yml
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: efs-claim
      securityContext:
        fsGroup: 0
      containers:
        - name: es
          image: elasticsearch:7.10.1
          ports:
            - name: http
              containerPort: 9200
            - containerPort: 9300
              name: inter-node
          volumeMounts:
            - name: es-config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              subPath: elasticsearch.yml
            - name: persistent-storage
              mountPath: /usr/share/elasticsearch/data

Answer 1

I encountered the same issue. GitHub copilot fixed it. So mainly you need to set up the folder permission first. a init container can do that:

          initContainers:
        - name: init-permissions
          image: busybox
          command: [ 'sh', '-c', 'chown -R 1000:1000 /es-data && chmod -R 775 /es-data' ]
          securityContext:
            privileged: true
          volumeMounts:
            - name: elasticsearch-data
              mountPath: /es-data

Answer 2

After some debugging I found out that the solution is not only to specify the fsGroup but also runAsUser and runAsGroup.

securityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsGroup: 1000