CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3
zabop
zabop

Reputation: 7852

How to generate URL to download file from S3 bucket

I would like to obtain URLs pointing to cloud-optimized geoTIFFs from Amazon's Copernicus Digital Elevation Model bucket.

After installing boto3 (with pip3 install boto3), I do, relying on this answer to the question Can I use boto3 anonymously? to download a single file:

import boto3
from botocore import UNSIGNED
from botocore.client import Config

s3 = boto3.client('s3', region_name='eu-central-1', config=Config(signature_version=UNSIGNED))

Then I query for list of objects in the bucket, using the second line of this answer to the question Use boto3 to download from public bucket:

objects = s3.list_objects(Bucket='copernicus-dem-30m')

I then access to a value in objects['Contents'], the first one, for example (ie index 0):

key = objects['Contents'][0]['Key']

key is now:

Copernicus_DSM_COG_10_N00_00_E006_00_DEM/Copernicus_DSM_COG_10_N00_00_E006_00_DEM.tif

I download this file by doing:

s3.download_file('copernicus-dem-30m', key, key.split('/')[-1])

Instead of downloading, how can I generate a URL, which later I can use to download the file, maybe using wget or just pasting it to a browswer?

This code shown above is based on the thread: How to get Copernicus DEM GeoTIFFs for a bounding box using Python.

Upvotes: 1

Views: 9047

Answers (3)

abdullah zulfiqar
abdullah zulfiqar

Reputation: 25

I had a secured view ad in that I did this:

@login_required
def product_download(request):
    product_id = request.GET.get('product_id')
    product = Product.objects.get(pk=product_id)
    if ProductOrder.objects.filter(order__user=request.user, product=product_id).exists():
        pre_signed_url = generate_presigned_url(product.excel_file)
        if pre_signed_url is None:
            return HttpResponse("Unable to generate download link.", status=500)
        return JsonResponse({"url":pre_signed_url})
    return HttpResponse("Unable to locate your purchased of the file.", status=500)

and then generate_presigned_url method. I am giving static key currently but you can save key of your file in every product and pass it as generic. solution above returns a url and opening that URL will automatically download the object.

import boto3
from django.conf import settings


def generate_presigned_url(file_key, expiration=60):
    """
    Generate a pre-signed URL for an Amazon S3 file.
    
    :param file_key: The key of the file in the S3 bucket
    :param expiration: Time in seconds for the pre-signed URL to remain valid
    :return: Pre-signed URL as string. If error, returns None.
    """
    s3_client = boto3.client('s3',
                                region_name=settings.AWS_REGION_NAME,
                                aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
                                aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY
                            )
    try:
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={'Bucket': settings.AWS_STORAGE_PRODUCTS_BUCKET_NAME,
                                                            'Key': "EW_9-Box Talent-Grid Template-v1.0.xlsx"},
                                                    ExpiresIn=expiration)
    except Exception as e:
        print(f"Error generating pre-signed URL: {e}")
        return None
    return response

Upvotes: 0

Geoffrey Burdett
Geoffrey Burdett

Reputation: 1976

S3 uses this format: https://<bucket-name>.s3.amazonaws.com/<key>

url = 'https://copernicus-dem-30m.s3.amazonaws.com/' + key

So the example above will look like this: https://copernicus-dem-30m.s3.amazonaws.com/Copernicus_DSM_COG_10_N00_00_E006_00_DEM/Copernicus_DSM_COG_10_N00_00_E006_00_DEM.tif

Upvotes: 4

Konrad Rudolph
Konrad Rudolph

Reputation: 545618

See Geoffrey’s answer for the format of the S3 URLs for public access buckets.

To generate a URL that works regardless of whether the bucket/object is public, you can use generate_presigned_url:

s3.generate_presigned_url(
    'get_object',
    Params = {'Bucket': 'copernicus-dem-30m', 'Key': key},
    ExpiresIn = SIGNED_URL_TIMEOUT
)

… with a suitably chosen SIGNED_URL_TIMEOUT (in seconds).

Upvotes: 4

Related Questions