CODEWITHSUNDEEP
amazon-s3boto3pre-signed-url
infinity_coding7
infinity_coding7

Reputation: 509

Boto3 generate presinged url does not work

Here is my code that I use to create a s3 client and generate a presigned url, which are some quite standard codes. They have been up running in the server for quite a while. I pulled the code out and ran it locally in a jupyter notebook

def get_s3_client():
    return get_s3(create_session=False)


def get_s3(create_session=False):
    session = boto3.session.Session() if create_session else boto3
    S3_ENDPOINT = os.environ.get('AWS_S3_ENDPOINT')

    if S3_ENDPOINT:
        AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
        AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
        AWS_DEFAULT_REGION = os.environ["AWS_DEFAULT_REGION"]

        s3 = session.client('s3',
                    endpoint_url=S3_ENDPOINT,
                    aws_access_key_id=AWS_ACCESS_KEY_ID,
                    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                    region_name=AWS_DEFAULT_REGION)

    else:
        s3 = session.client('s3', region_name='us-east-2')

    return s3

s3 = get_s3_client()

BUCKET=[my-bucket-name]
OBJECT_KEY=[my-object-name]


signed_url = s3.generate_presigned_url(
    'get_object',
    ExpiresIn=3600,
    Params={
        "Bucket": BUCKET,
        "Key": OBJECT_KEY,
    }
)

print(signed_url)

When I tried to download the file using the url in the browser, I got an error message and it says "The specified key does not exist." I noticed in the error message that my object key becomes "[my-bucket-name]/[my-object-name]" rather than just "[my-object-name]".

Then I used the same bucket/key combination to generate a presigned url using aws cli, which is working as expected. I found out that somehow the s3 client method (boto3) inserted [my-object-name] in front of [my-object-name] compared to the aws cli method. Here are the results

From s3.generate_presigned_url()

https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-bucket-name]/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAV17K253JHUDLKKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T175014Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=5cdcc38e5933e92b5xed07b58e421e5418c16942cb9ac6ac6429ac65c9f87d64

From aws cli s3 presign

https://[my-bucket-name].s3.us-east-2.amazonaws.com/[my-object-name]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAYA7K15LJHUDAVKHB%2F20210520%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210520T155926Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=58208f91985bf3ce72ccf884ba804af30151d158d6ba410dd8fe9d2457369894

I've been working on this and searching for solutions for day and half and I couldn't find out what was wrong with my implementation. I guess it might be that I ignored some basic but important settings to create a s3 client using boto3 or something else. Thanks for the help!

Upvotes: 0

Views: 1021

Answers (1)

infinity_coding7
infinity_coding7

Reputation: 509

Ok, myth is solved, I shouldn't provide the endpoint_url=S3_ENDPOINT param when I create the s3 client, boto3 will figure it out. After i removed it, everything works as expected.

Upvotes: 1

Related Questions