Unable to load AWS credentials on EC2 Instance

Question

The application

Simple REST API registration service in Spring, after sending proper POST request new user is created in database and Amazon SES sends an email with registration link to verify.

The problem

Locally after setting local variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION) in my OS (Windows) app works just fine, but the problem starts after deploying it. I have an EC2 Instance with Amazon Linux AMI on AWS:

• created user at AWS Identity and Access Management (IAM), granted AmazonSESFullAccess role
• logging by CMD using shh with the private key file *.pem
• Tomcat8 service started
• MySQL service started
• application *.war file deployed
• created environment variables using 'export' command and I checked 'printenv' just to be sure everything is fine
• after sending POST request I got exception (below) which means that user has been created but Amazon SES didn't send an email confirmation because couldn't authenticate

{
    "timestamp": "2020-04-26T15:44:44.010+0000",
    "message": "Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@23fac1a3: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@68aa5a98: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]"
}

• I checked again local environment variables on my EC2 instance and it was looking fine but to be sure I re-configured it using 'aws configure' command

• exception keeps showing, somehow application cannot get environment variables, I'm fighting with that for over 5 hours now so hopefully someone will come here to rescue me...

Piece of code (works fine locally):

        AmazonSimpleEmailService client =
                AmazonSimpleEmailServiceClientBuilder
                        .standard()
                        .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
                        .withRegion(Regions.EU_CENTRAL_1)
                        .build();

I am total Linux noob, having problems with simple commands so please be gentle with solutions requiring some console commands.

Answer 1

If you're running app on EC2, don't use IAM user.

Instead create IAM role with same permissions and assign that role to the instance. If app uses AWS SDK it will be able to pick up credentials without any problems.

In your case problem is probably app's environment being different from yours, if you export credentials in your bash session it will not pass to app if it's loaded under different user or bash session.

Answer 2

The DefaultAWSCredentialsProvider has multiple places it will look for credentials. Instead of setting up your credentials as an environment variable, you can set up a credentials profile. See this documentation: Working with AWS Credentials.

Make sure you have the AWS CLI installed, then you can run the following command to configure your profile: aws configure

Click here for the documentation on the aws configure command.

If you have already configured your aws profile and it still does not work, you have most likely configured the profile for the wrong linux user. For example, if a linux user named tomcat8 is the user who is running your tomcat instance, then you need to set up a credentials profile at /home/tomcat8/.aws/credentials/