CODEWITHSUNDEEP
ubuntupermissionsmountamazon-efs
Michael M
Michael M

Reputation: 8733

EFS mount failing with mount.nfs4: access denied by server

Trying to mount an EFS file system. The only thing I changed was removing the default SG created with the EFS group and replacing it with a custom SG that my EC2 instances are already in.

AWS provides the necessary command for mounting the NFS share and it SHOULD work verbatim. Often it does. But sometimes you get this:

mount.nfs4: access denied by server while mounting fs-xxxxxxxx.efs.us-west-2.amazonaws.com:/

Sadly, the troubleshooting documentation says under the heading "Action to Take":

If you are attempting to mount the file system using IAM...

... and has absolutely zero recommendation for what to do it your are NOT attempting to mount the FS using IAM.

Now in the first place, I am quite sure I am not doing something wrong because I have playbooks that I've used dozens of times to mount EFS (NFS) shares to EC2 instances are they are quite polished by now. So why would it sometimes fail?

Upvotes: 8

Views: 30289

Answers (7)

suren
suren

Reputation: 8786

I was getting this error mounting an EFS access point into a container (FARGATE).

Permissions-wise, I do not have the permissions mentioned in other answers. All I have is the task execution pole, that is required and is bound an aws predefined policy AmazonECSTaskExecutionRolePolicy, and a task policy that I have defined to write on log stream, execute commands in the container, kms, etc. No EFS related permissions.

What I was doing wrong is that I was not defining Root directory creation permissions. This permission is used to create the Root directory path within EFS. So the access denied in my case meant that the directory was not yet there and I was trying to mount it in the container.

These permissions are not necessary if someone is going to mount filesystems root directory. Actually, you probably shouldn't be using an access point to do that, but other than the root directory, you need it.

In fact, as a test, if one doesn't define Root directory creation permissions (therefor gets the access denied error), can mount the filesystem's root directory and manually create the necessary path, and then try to mount it, it will work just fine.

Upvotes: 0

Daniel Draper
Daniel Draper

Reputation: 11

Agree with Michael M's solution. We had identical issue after creating volumes with default SG and then moving the mount targets to a specific EFS SG. After two days of troubleshooting I finally dropped the volumes and recreated them, using the EFS SG at the time of creation. Worked without issue after that.

Upvotes: 1

Pbd
Pbd

Reputation: 1319

I faced a similar issue and followed StartupGuy's steps. That didn't particularly fix my issue, so I traced the cloud trail events and realised that the access policy needs to have mount access perms as well.

This is the default action for fs-policy:

             ...
             "Action": [
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:ClientWrite"
            ]
            ...

You need to add "elasticfilesystem:ClientMount" as well to the fs policy.

Upvotes: 16

Coop Weisbach
Coop Weisbach

Reputation: 105

If you have a File System Policy set and you are using 'sudo mount -t nfs' or 'nfs4,' then you get this error. You can either:

  • A.) Remove the file system policy from the file system
  • B.) Use the following command to mount: sudo mount -t efs -o tls,iam ...

Source: https://docs.aws.amazon.com/efs/latest/ug/mounting-IAM-option.html

Upvotes: 3

Nathan Zook
Nathan Zook

Reputation: 31

For me, the problem was that I had a policy requiring encryption in transit for the drive, and the instance creation wizard creates a bad /etc/fstab entry.

  1. The policy requires that the drive be mounted with tls. Instructions for this are given here: https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-helper-ec2-linux.html, IF you use the mount helper, and specify -o tls.
  2. The /etc/fstab created by the instance creation wizard does not perform the proper mount. In fact, the "Using the NFS client" option on that same page is equivalent to the bad entry which is created.

Here is what a proper /etc/fstab entry looks like for encryption in transit:

fs-0123456789abcdef0:/ /mnt/fs-1 efs tls,_netdev 0 0

Upvotes: 1

HeikoG
HeikoG

Reputation: 991

I got this error message when the directory on the EFS did not exist while trying to access through an access point.

Upvotes: 8

Michael M
Michael M

Reputation: 8733

Well as it turns out, AWS is not always as slick as it usually feels and sometimes things get botched on the back-end. In this case, it was possible that replacing the SG actually might have appeared to work in the UI, but on the back-end did not take effect. I'm just guessing.

What I did was just delete the existing EFS and create the exact same EFS. This time the only thing I did different was set my custom EFS SG on creation of the EFS, instead of replacing it after creation and viola, my playbooks were working again.

There is absolutely no intuitive (or documented) reason that I can think of, why starting with non-default SG should be any different than replacing it, when its the exact same SG. In either case you are setting up the EFS to use your selected SG and EFS is not objecting. Besides, I've done that before with no problems. So I am chalking this up as a EFS/SG back-end screw up that wasted a lot of time to troubleshoot.

In summary, if a new EFS share is giving you the mount.nfs4: access denied by server error when trying a standard mount (and you know you are doing everything else correctly) - just delete it and just re-create it. Don't necessarily assume you are doing something wrong and that AWS services can't screw up now and then.

Upvotes: 9

Related Questions