CODEWITHSUNDEEP
amazon-web-servicesamazon-s3wget
Nikhil Mittal
Nikhil Mittal

Reputation: 1

Download a folder from S3 bucket using wget

I have made my S3 bucket public for certain IP. In the bucket policy, I have given s3:GetObject permission.

Now I am trying to download a folder from that bucket using wget. Example:

wget -r "https://....."

But getting response: HTTP request sent, awaiting response...

403 Forbidden

But, when I am trying to download a single file from the folder, I am able to download it without any issue.

Getting response:

HTTP request sent, awaiting response...

403 Forbidden

I expected the folder to be downloaded

Upvotes: 0

Views: 846

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269091

It is not possible to "download a folder" in one request. You must retrieve each object individually.

The easiest way to download a folder is to use the AWS Command-Line Interface (CLI). You can either use:

aws s3 sync s3://bucket-name/foldername <name-of-local-dir>

or:

aws s3 cp --recursive s3://bucket-name/foldername <name-of-local-dir>

The sync command only copies files that do not already exist, while cp --recursive will copy all files. Both commands will also copy sub-folders.

Update:

To provide access without using AWS credentials, ensure that the bucket is publicly accessible to the IP desired address using a policy like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:ListObjects",
                "s3:GetObjects"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "1.2.3.4/32"
                }
            }
        }
    ]
}

You might need to provide extra permissions -- if the copy fails, the error message will tell you what it was trying to do.

You can then copy with:

aws s3 sync --no-sign-request s3://bucket-name/foldername <name-of-local-dir>

If this is performed from the nominated IP address, it should be able to download the files without AWS credentials.

Upvotes: 2

Related Questions