CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto3
treyhakanson
treyhakanson

Reputation: 4941

Bulk Generate Pre-Signed URLs boto3

I am currently using the following to create a pre-signed url for a bucket resource:

bucket_name = ...
key = ...
s3_client = ...

s3_client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Key": key
    },
    ExpiresIn=100
)

This works fine. However, I was wondering if it was possible to generate pre-signed urls for multiple keys in one request? Or is it required to make one request for each key? I didn't find anything useful in the docs regarding this topic. I'm looking for something like this:

bucket_name = ...
keys = [...]
s3_client = ...

# Returns an array of pre-signed urls, in the same order as `keys`
s3_client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Keys": keys
    },
    ExpiresIn=100
)

Upvotes: 16

Views: 6469

Answers (2)

dp6000
dp6000

Reputation: 673

As @John Rotenstein mentioned in his response, you can repeatedly call this function inside a For Loop.

Here is an example:

def create_presigned_urls(s3Client, bucket_name: str, key: str, expires_in: int):
  """Create presigned_urls
  Args:
      s3Client (s3 Class): boto3 S3 Class
      bucket_name
      key
      expires_in: The number of seconds the presigned URL is valid for.

  Returns:
      (string): presigned URL
  """
  presigned_url = s3Client.generate_presigned_url(
    ClientMethod="get_object",
    Params={
        "Bucket": bucket_name,
        "Key": key
    },
    ExpiresIn=expires_in
  )
  return presigned_url

For loop:

s3Client = boto3.client('s3')
bucket_name = 'BUCKET_NAME'
expires_in = 3600
list_file_url = []

for index, unit in df.iterrows():
    key = df['key_name']
    url = create_presigned_urls(s3Client, bucket_name, key, expires_in)
    list_file_url.append(url)

Upvotes: 1

John Rotenstein
John Rotenstein

Reputation: 270294

Generating presigned URLs is actually done locally, without requiring a call to AWS. This is because all necessary information (Bucket, Key, Secret Key) is known locally and can generate the signature.

Therefore, feel free to call that function repeatedly since there is no network/service overhead.

In general, there should be no need to bulk-generate URLs. Instead, whenever your application wishes to reference a resource (eg an image on an HTML page), it can quickly call the generate_presigned_url() function with an appropriate timeout.

Upvotes: 40

Related Questions