CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto3
Lewis Saunders
Lewis Saunders

Reputation: 91

Trouble downloading S3 bucket objects through boto3. Error 403 HeadObject: Forbidden

I'm aware there are other threads on here about this issue but am still struggling to find the right solution. I am attempting to download a set of specific objects within an S3 bucket (that I do have access to) using the following python script. When running the script, the first object successfully downloads but then this error (403) is thrown:

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

See below my code:

import csv
import boto3
import re
import logging
from botocore.exceptions import ClientError

prod_number_array_bq = []
prod_number_array_s3 = []
with open('bq-results-20191218-151637-rshujisvqrri.csv') as csv_file:
    csv_reader = csv.reader(csv_file,delimiter=',')
    line_count = 0
    for row in csv_reader:
        sliced = re.sub("[^0-9]", "", str(row))
        prod_number_array_bq.append(sliced)

s3 = boto3.resource('s3')
bucket = s3.Bucket('********')

for key in bucket.objects.all():
    sliced = re.sub("[^0-9]", "", str(key.key))
    if((set(sliced) & set(prod_number_array_bq))!=""):
            bucket.download_file(key.key,sliced + '.txt')

Help would be appreciated :)

Thanks

Upvotes: 5

Views: 10399

Answers (2)

Areza
Areza

Reputation: 6080

In my case, I could read the file but couldn't download it

So I the following would have printed the file information

resp = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=origin)
print(resp)

but then this would have given botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden error

s3_client.download_file(bucket_name, origin, destination)

The problem was the model was uploaded from different AWS account. We were missing ACL on upload. so,

so we uploaded the file with the following command

s3_client.upload_file(origin,
                      bucket_name,
                      destination,
                      ExtraArgs={'ACL':'bucket-owner-full-control'})

and this led us to read and download the file as we expectd.

Upvotes: 3

Ngenator
Ngenator

Reputation: 11259

Typically when you see a 403 on HeadObject despite having the s3:GetObject permission, it's because the s3:ListObjects permission wasn't provided for the bucket AND your key doesn't exist. It's a security measure to prevent exposing information about what objects are or aren't in your bucket. When you have both the s3:GetObject permission for the objects in a bucket, and the s3:ListObjects permission for the bucket itself, the response for a non-existent key is a 404 "no such key" response. If you only have s3:GetObject permission and request a non-existent object, the response is a 403 "access denied".

Upvotes: 7

Related Questions