CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto3
Jesse Reza Khorasanee
Jesse Reza Khorasanee

Reputation: 3491

How to create a signed s3 url for requester pays bucket in python

I have a requester pays bucket that I do not control in form:

s3://bucket-name/path-to-my-file

I am attempting to generate a presigned url to send to a web app to render it in browser.

I've gone through the boto s3 documentation but can't find anything that covers this :(

My script below creates returns URL that does not have access and returns this error from s3:

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>11DCA24D8DF2E9E8</RequestId>
  <HostId>SeTDlt66hPsj5/dV1pOT9GnYyWgpSGI4ezI3wC7iz8Pny9sy2sUWsuUsl4JkEQeBXXIyiE8EXCk=</HostId>
</Error>

I'm confident this is because the bucket is requester pays, becuase when I run this command in aws cli it works:

aws s3 cp s3://blackfynn-discover-use1/66/2/files/Derivatives . --request-payer requester --recursive

But this one returns Forbidden:

aws s3 cp s3://blackfynn-discover-use1/66/2/files/Derivatives .

Here's my python script which would work if it was not requester pays:

import requests
import boto3

def get_signed_url(s3_url):
    # Get the service client.
    s3 = boto3.client('s3')
    bucket_name, key_name = split_s3_bucket_key(s3_url)
    # Generate the URL to get 'key-name' from 'bucket-name'
    url = s3.generate_presigned_url(
        ClientMethod='get_object',
        Params={
            'Bucket': bucket_name,
            'Key': key_name
        }
    )
    return url

def split_s3_bucket_key(s3_path):
    """Split s3 path into bucket and key prefix.
    This will also handle the s3:// prefix.
    :return: Tuple of ('bucketname', 'keyname')
    """
    if s3_path.startswith('s3://'):
        s3_path = s3_path[5:]
    return find_bucket_key(s3_path)

def find_bucket_key(s3_path):
    """
    This is a helper function that given an s3 path such that the path is of
    the form: bucket/key
    It will return the bucket and the key represented by the s3 path
    """
    s3_components = s3_path.split('/')
    bucket = s3_components[0]
    s3_key = ""
    if len(s3_components) > 1:
        s3_key = '/'.join(s3_components[1:])
    return bucket, s3_key

s3_file_path = 's3://blackfynn-discover-use1/66/2/files/Derivatives/manifest.xlsx'
get_signed_url(s3_file_path)

Upvotes: 2

Views: 1600

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270294

It looks like the URL will need to include x-amz-request-payer=requester, but this might also need to be specified when creating the pre-signed URL.

Try the advice shown below, then let us know whether it worked for you!

From Downloading Objects in Requester Pays Buckets - Amazon Simple Storage Service:

For signed URLs, include x-amz-request-payer=requester in the request

From Support Requester Pays S3 buckets · Issue #346 · samtools/htslib:

OK, was able to compile htslib with good libcurl support. Confirmed that it can take a presigned URL to view files:

import boto3
client = boto3.client('s3')
url  = client.generate_presigned_url("get_object", Params={"Bucket":"angel-reqpay","Key":"test.cram" , "RequestPayer":'requester'})

From AWS Developer Forums: Announcing “Requester Pays” Option for ...:

You URL would look something like:

http://somebucket.s3.amazonaws.com/key/[.....]&x-amz-request-payer=requester

Upvotes: 4

Related Questions