Cannot write files on mounted volume aws efs

Question

I'm building a scalable and distributed architecture for workflows within our company by leveraging the use of Apache Airflow. I'm using ECS Fargate.

Since this is a distributed architecture, a non-centralized file system is required in order to have a consistent-shared view across all machines (ie, the webserver needs to access the DAGs files, scheduler, and workers also).

For such purpose, I'm using AWS EFS, I can successfully mount the file system into an EC2 instance, but I'm unable to write or create a file within it.

This is the policy attached to the fs:

{
    "Version": "2012-10-17",
    "Id": "ExamplePolicy01",
    "Statement": [
        {
            "Sid": "ExampleSatement01",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "elasticfilesystem:ClientMount",
                "elasticfilesystem:ClientWrite"
            ],
            "Resource": "arn:aws:elasticfilesystem:eu-west-1:XXXXXXXXXX:file-system/fs-aaaaaaa"
        }
    ]
}

and as the doc says:

elasticfilesystem:ClientWrite ------> Provides an NFS client with write permissions on a file system.

I'm mounting by executing the following command:

 sudo mount -t efs -o tls,accesspoint=fsap-oooooooooooooooo fs-aaaaaaa:/ efs

and indeed, I can see the content, but got Permission denied every time I try to write.

[ec2-user@ip-XXXX ~]$ sudo mount -t efs -o tls,accesspoint=fsap-oooooooooooooooo fs-aaaaaaa:/ efs
[ec2-user@ip-XXXX ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  616K  2.0G   1% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/xvda1      8.0G  2.7G  5.4G  33% /
tmpfs           395M     0  395M   0% /run/user/1000
tmpfs           395M     0  395M   0% /run/user/0
127.0.0.1:/     8.0E     0  8.0E   0% /home/ec2-user/efs
[ec2-user@ip-XXXX ~]$
[ec2-user@ip-XXXX ~]$ cd efs/
[ec2-user@ip-XXXX efs]$ touch a
touch: cannot touch ‘a’: Permission denied
[ec2-user@ip-XXXX efs]$
[ec2-user@ip-XXXX efs]$ sudo touch a
touch: cannot touch ‘a’: Permission denied
[ec2-user@ip-XXXX efs]$ sudo su -
Last login: Wed Aug 19 21:29:39 UTC 2020 on pts/0
[root@ip-XXXX ~]#
[root@ip-XXXX ~]# cd /home/ec2-user/efs/
[root@ip-XXXX efs]#
[root@ip-XXXX efs]# touch d
touch: cannot touch ‘d’: Permission denied

Any help would be much appreciated. Thank you.

Answer 1

I also encountered this problem. The solution that worked for me was two steps

1. Add elasticfilesystem:ClientRootAccess to the action statement of the policy. For example,

       "Action": [
           "elasticfilesystem:ClientRootAccess",
           "elasticfilesystem:ClientMount",
           "elasticfilesystem:ClientWrite"
       ],
   

2. Then, after mounting the EFS, change it's permissions to allow writing. For example if I mounted to /efs/data, then

   cd /efs/data
   sudo chmod 775 /efs/data
   

And now it works.

If you want to avoid step 2, you can go into the AWS "Access Point" settings and update the "POSIX User" and "Root directory creation permissions" to be your user/group IDs. For example, in ubuntu the default user usually has a POSIX ID of "1000" and a group ID of "1000".

To find out your user ID,

 id -u <username>

To find out your group ID,

 id -g <username>

Answer 2

I have successfully achieved the goal, and this answer is intended to share my experience just in case someone bumps into the same use case.

If you are using ECS Fargate, there is an option to mount volumes in the version with platform 1.4.0. So, within container_definitions there is an option called mountPoints:

    ...
    "entryPoint": [],
    "command": [],
    "portMappings": [
      {
        "hostPort": ${host_port},
        "protocol": "tcp",
        "containerPort": ${container_port}
      }
    ],
    "mountPoints": [
      {
        "sourceVolume": "dags",
        "containerPath": "/usr/local/airflow/dags",
        "readOnly": false
      }
    ],
    "cpu": ${cpu},
    "environment": [
      {
        "name": "POSTGRES_HOST",
        "value": "${postgres_host}"
      },
      {
        "name": "AIRFLOW_COMPONENT",
        "value": "webserver"
      },
      {
        "name": "REDIS_HOST",
        "value": "${redis_host}"
      }
    ]
    ...

that will mount the EFS volume. Remember to specify the volume configuration. Here's the terraform:

resource "aws_ecs_task_definition" "task_definition_airflow_webserver" {
  family                    = "airflow-webserver"
  requires_compatibilities  = ["FARGATE"]
  network_mode              = "awsvpc"
  execution_role_arn        = "${aws_iam_role.ecs_service_role_airflow_webserver.arn}"
  container_definitions     = "${data.template_file.airflow_webserver.rendered}"
  cpu                       = 1024
  memory                    = 2048

  volume {
    name = "dags"

    efs_volume_configuration {
      file_system_id          = "${aws_efs_file_system.airflow_dags_efs.id}"
      root_directory          = "/"
      transit_encryption      = "ENABLED"
    }
  }
}

Hope this helps. Let me know if there's something ambiguous or unclear.