Unable to copy from S3 to Ec2 instance

Question

I am trying to copy a file from S3 to an Ec2 instance, here is the strange behavior Following command runs perfectly fine and show me the contents of s3, that I want to access

$aws s3 ls
2016-05-05 07:40:57 folder1
2016-05-07 15:04:42 my-folder

then I issue following command (also successful)

$ aws s3 ls s3://my-folder
2016-05-07 16:44:50       6007 myfile.txt

but when I try to copy this file, I recive an error as follows

$aws s3 cp s3://my-folder/myfile.txt ./

A region must be specified --region or specifying the region in a configuration file or as an environment variable. Alternately, an endpoint can be specified with --endpoint-url

I simply want to copy txt file from s3 to ec2 instance. At least modify the above command to copy the contents. I am not sure about region as If I visit S3 from web it says

"S3 does not require region selection"

What is happening on the earth?

Answer 1

The fix is specifying the region, below explains how to get the bucket region if you cant get it from the cli.

aws s3 cp s3://xxxxyyyyy/2008-Nissan-Sentra.pdf myfile.pdf --region us-west-2

Answer 2

How about

aws s3 cp s3://my-folder/myfile.txt .
# or 
aws s3 cp s3://my-folder/myfile.txt myfile.txt

I suspect the problem is something to do with the local path parser. aws cli s3 fileformat parser

It is kinda strange because aws cli read the credential and region config.

Answer 3

Most likely something is not working right, you should not be able to list the bucket if your regions is not setup as default in the aws configure.

Therefore from my experience with S3 if this works:

aws s3 ls s3://my-folder

then this should work as well:

aws s3 cp s3://my-folder/myfile.txt ./

However if it's asking you for region, then you need to provide it.

Try this to get the bucket region:

aws s3api get-bucket-location --bucket BUCKET

And then this to copy the file:

aws s3 cp --region <your_buckets_region> s3://my-folder/myfile.txt ./

If I visit S3 from web it says

"S3 does not require region selection"

S3 and bucket regions can be very confusing especially with that message. As it is the most misleading information ever IMO when it comes to s3 regions. Every bucket has got specific region (default is us-east-1) unless you have enabled cross-region replication.

You can choose a region to optimize latency, minimize costs, or address regulatory requirements. Objects stored in a region never leave that region unless you explicitly transfer them to another region. For more information about regions, see Accessing a Bucket: in the Amazon Simple Storage Service Developer Guide.