Proper credentials for node & s3

Question

My goal is to set up an S3 bucket that my node application can download objects from. I am new to building websites and api requests and have spent a lot of time reading up on AWS's documentation but am really confused.

What are the best ways to set up credentials so that my node application can connect to S3 and download the specific objects I need? Should I be using an iAm role? Are the iAm access keys then just stored on my local .aws/credentials file? So far I have been trying to run the code below with first my personal aws user's access keys, and then an iAm role's access keys, saved into my .aws/credentials file, but I keep getting Access Denied (stack trace below). Am I not setting up my credentials correctly? Thanks!

var AWS = require('aws-sdk');

var s3 = new AWS.S3();

var credentials = new AWS.SharedIniFileCredentials({profile: 'personal-account'});
AWS.config.credentials = credentials;

var params = { Bucket: "my_bucket", Key: "testing.txt" }

s3.getObject(params, function(err, data) {
  if (err) console.log(err, err.stack);
  else     console.log(data);
});

Stack Trace:

 { AccessDenied: Access Denied
at Request.extractError (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/services/s3.js:577:35)
at Request.callListeners (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
message: 'Access Denied',
code: 'AccessDenied',
region: null,
time: 2017-10-26T02:01:15.586Z,
requestId: '10371DCDBBC02508',
extendedRequestId: '0GC8BZ/39/eFOzWgTedHSFxhFSGBAMcZqxCVAlUxp8YamwBGeGZUZVe7Ti9O/6+BxhUTk9jb4hk=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 75.06631783335284 } 'AccessDenied: Access Denied
at Request.extractError (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/services/s3.js:577:35)
at Request.callListeners (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:683:14)    
at Request.transition (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:22:10)    
at AcceptorStateMachine.runTo (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/state_machine.js:26:10    
at Request.<anonymous> (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:38:9)    
at Request.<anonymous> (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/request.js:685:12)    
at Request.callListeners (/Users/jemery62/dev/jimmyemery_com/node_modules/aws-sdk/lib/sequential_executor.js:115:18)'

UPDATE:

Hi, I am still unable to get access to this file. My .aws/credentials looks like the following (I am using my personal-account profile, which pulls my IAm role):

[default] ; default profile
aws_access_key_id = defaultAccessKey
aws_secret_access_key = defaultSecretAccessKey
region = us-east-1

[personal-account] ; personal account profile
aws_access_key_id = personalAccessKey
aws_secret_access_key = personalSecretAccessKey
region = us-east-1

.aws/config:

[profile corp]
role_arn = corp-role-arn
region = us-east-1
source_profile = default

I'm not sure I understand too well how IAm policies work, but I have my IAm role set up with AdministratorAccess and the following profile:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Do I need to make additional changes to my .aws/config? Am I resourcing my credentials incorrectly or incorrectly trying to call the object I want?

UPDATE 2:

I have posted a second question here to clarify exactly the issue I am experiencing at this moment, as I think this post I have made is starting to become a bit confusing.

Answer 1

The code below ended up working for me. It looks like the I needed to instantiate s3 after I configured AWS.

var AWS = require('aws-sdk');

var credentials = new AWS.SharedIniFileCredentials({profile: 'personal-account'});
AWS.config.credentials = credentials;

s3 = new AWS.S3();

var params = { Bucket: "my_bucket", Key: "testing.txt" }

s3.getObject(params, function(err, data) {
  if (err) console.log(err, err.stack);
  else     console.log(data);
});

Answer 2

I personally found that loadFromPath method to load credentials stored in a JSON formatted file result the most flexible way to do it.

You need indeed for an IAM role (choose carefully it access scope) to obtain both the accessKeyId and you secretAccessKey

So your JSON file should look like this:

{
  "accessKeyId": "IAM_ACCESS_KEY_ID",
  "secretAccessKey": "IAM_SECRET_ACCESS_KEY"
}

Then you can simply do:

var AWS = require('aws-sdk');
AWS.config.loadFromPath('/path/to/credentials/aws.credentials.json');

var params = { Bucket: "my_bucket", Key: "testing.txt" }

s3.getObject(params, function(err, data) {
  if (err) console.log(err, err.stack);
  else     console.log(data);
});

Answer 3

If your code is running on EC2 then it should be using an EC2 instance profile. If it is running anywhere else then you should have an IAM user's keys configured in either ~/.aws/credentials or in environment variables.

In either case, the IAM role or user should have an attached policy that gives it access to the S3 bucket and the objects within that bucket. If you have further issues then add the IAM policy you are using to your question.