AWS boto3 - InvalidToken Error when calling the GetObject operation

Question

I'm currently trying to invoke a lambda function locally through the sam local invoke command, but I keep getting the InvalidToken error with the GetObject operation.

What I've tried:

• Since I'm operating everything on ap-northeast-2 region, I double-checked that my config file .aws folder was set to the same region.
• I've tried printing (although not recommended) the credentials through the boto3 Session and confirmed that the credentials used are the same as the one in the credentials folder (the only credential there)
• I've tried adding the --region ap-northeast-2 flag in my invocation command, but it gave the same error.
• I've checked the current user has full access to S3 resources (it has an Admin role). I've also tried detaching & reattaching roles (deleting the config & credentials files and running aws configure again with another admin role) but had no luck.

Could there be anything else that I might have forgotten? Below are the code and the log output. Thanks in advance :)

# imports omitted
logger.info('Starting...')
s3 =  boto3.client('s3')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

session = boto3.Session()
credentials = session.get_frozen_credentials()
print(credentials.access_key, credentials.secret_key, credentials.token)

def load_model(): # handler function 
    logger.info('Loading model from S3')
    obj = s3.get_object(Bucket=MODEL_BUCKET, Key=MODEL_KEY )
    # obj = s3.Object(bucket_name='ml-project-paul-kang', key='fastai-models/lesson1/model.tar.gz')
    logger.info('Model loading complete')
    bytestream = io.BytesIO(obj['Body'].read()) # further code omitted

And this is what I get after running sam local invoke PyTorchFunction -n env.json -e event.json:

Invoking app.lambda_handler (python3.6)
arn:aws:lambda:ap-northeast-2:<ID>:layer:pytorch-p36:1 is already cached. Skipping download
Requested to skip pulling images ...

[INFO]  2020-08-22T08:18:22.685Z                Starting...
<CURRENT_USER_ACCESS_KEY> <CURRENT_USER_SECRET_KEY> (null)
[INFO]  2020-08-22T08:18:22.685Z                Model Bucket is ml-project
[INFO]  2020-08-22T08:18:22.685Z                Model Prefix is fastai-models/lesson1/model.tar.gz
[INFO]  2020-08-22T08:18:22.686Z                Loading model from S3
module initialization error: An error occurred (InvalidToken) when calling the GetObject operation: The provided token is malformed or otherwise invalid.

Answer 1

I think your s3 bucket is not located in ap-northeast-2 region. You can check this on the s3 console. if its not you will need to pass the region in the code as below.

s3 =  boto3.client('s3', region_name='ap-northeast-2')