CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3boto3
user6248190
user6248190

Reputation: 1271

How to copy all files from one folder to another folder in the same S3 bucket

I am trying to copy all files from one folder to another folder in the s3 bucket. I see a lot of examples for moving from one bucket to another but not from one folder to another.

from the examples I have seen online, I have written the following code.

import boto3
s3_resource = boto3.resource('s3')
src = 'old_folder'
dest = 'new_folder'

for key in s3_resource.list_objects(Bucket=config.S3_BUCKET)['Contents']:
    files = key['Key']
    copy_source = {'Bucket': config.S3_BUCKET, 'Key': f'{src}/{files}'}
    s3_resource.meta.client.copy(
        copy_source, f'{config.S3_BUCKET}/{dest}/', f'{src}/{files}')

however, when I run the code, I get the following error:

AttributeError: 's3.ServiceResource' object has no attribute 'list_objects'

for the following line: for key in s3_resource.list_objects(Bucket=config.S3_BUCKET)['Contents']:

I believe I am getting the error because I am using boto3.resource(s3) instead of boto3.client(s3) however the examples I have seen online seem to be using boto3.resource(s3) to move files from one bucket to another.

What is the correct approach to take to move all files from one folder to another folder in the same bucket in s3 using python?

Upvotes: 1

Views: 2954

Answers (1)

Marcin
Marcin

Reputation: 238199

I modified the code and also added pagination. Please have a look at it:

s3 = boto3.client('s3')

src = 'old_folder'
dest = 'new_folder'

paginator = s3.get_paginator('list_objects')

operation_parameters = {'Bucket': config.S3_BUCKET,
                        'Prefix': src}

page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:
    for obj in page['Contents']:
        file = obj['Key']
        #print(file)    
        dest_key = file.replace(src, dest)
        #print("dest_key)
        s3.copy_object(Bucket=config.S3_BUCKET, 
                       CopySource=f'/{config.S3_BUCKET}/{file}', 
                       Key=dest_key)

Upvotes: 1

Related Questions