CODEWITHSUNDEEP
amazon-web-servicesamazon-s3boto3
asmd ashdkj
asmd ashdkj

Reputation: 133

Is there a simple way to rename s3 folder via boto3?

I have s3 bucket with folder, and inside the folder there are large files.

I want to rename the folder with python3-boto3 script.

I read this ("How to Rename Amazon S3 Folder Objects with Python"), and what he is doing is to copy the files with new prefix, then deleting the original folder.

It is very not efficient way to do it, and because I have large files, it will take long time to do it.

Is there a simpler/more efficient way to do it?

Upvotes: 2

Views: 6882

Answers (2)

ViaTech
ViaTech

Reputation: 2813

Simple, no. Unfortunately.

There are a lot of 'issues' with folder structures in s3 it seems as the storage is flat.

I have a Django project where I needed the ability to rename a folder but still keep the directory structure in-tact, meaning empty folders would need to be copied and stored in the renamed directory as well.

aws cli is great but neither cp or sync or mv copied empty folders (i.e. files ending in '/') over to the new folder location, so I used a mixture of boto3 and the aws cli to accomplish the task.

More or less I find all folders in the renamed directory and then use boto3 to put them in the new location, then I cp the data with aws cli and finally remove it.

import threading

import os
from django.conf import settings
from django.contrib import messages
from django.core.files.storage import default_storage
from django.shortcuts import redirect
from django.urls import reverse

def rename_folder(request, client_url):
    """
    :param request:
    :param client_url:
    :return:
    """
    current_property = request.session.get('property')
    if request.POST:
        # name the change
        new_name = request.POST['name']
        # old full path with www.[].com?
        old_path = request.POST['old_path']
        # remove the query string
        old_path = ''.join(old_path.split('?')[0])
        # remove the .com prefix item so we have the path in the storage
        old_path = ''.join(old_path.split('.com/')[-1])
        # remove empty values, this will happen at end due to these being folders
        old_path_list = [x for x in old_path.split('/') if x != '']

        # remove the last folder element with split()
        base_path = '/'.join(old_path_list[:-1])
        # # now build the new path
        new_path = base_path + f'/{new_name}/'
        # remove empty variables
        # print(old_path_list[:-1], old_path.split('/'), old_path, base_path, new_path)
        endpoint = settings.AWS_S3_ENDPOINT_URL
        # # recursively add the files
        copy_command = f"aws s3 --endpoint={endpoint} cp s3://{old_path} s3://{new_path} --recursive"
        remove_command = f"aws s3 --endpoint={endpoint} rm s3://{old_path} --recursive"
        
        # get_creds() is nothing special it simply returns the elements needed via boto3
        client, resource, bucket, resource_bucket = get_creds()
        path_viewing = f'{"/".join(old_path.split("/")[1:])}'
        directory_content = default_storage.listdir(path_viewing)

        # loop over folders and add them by default, aws cli does not copy empty ones
        # so this is used to accommodate
        folders, files = directory_content
        for folder in folders:
            new_key = new_path+folder+'/'
            # we must remove bucket name for this to work
            new_key = new_key.split(f"{bucket}/")[-1]
            # push this to new thread
            threading.Thread(target=put_object, args=(client, bucket, new_key,)).start()
            print(f'{new_key} added')

        # # run command, which will copy all data
        os.system(copy_command)
        print('Copy Done...')
        os.system(remove_command)
        print('Remove Done...')

        # print(bucket)
        print(f'Folder renamed.')
        messages.success(request, f'Folder Renamed to: {new_name}')

    return redirect(request.META.get('HTTP_REFERER', f"{reverse('home', args=[client_url])}"))

Upvotes: 2

E.J. Brennan
E.J. Brennan

Reputation: 46841

There is no way to rename s3 objects/folders - you will need to copy them to the new name and delete the old name unfortunately.

There is a mv command in the aws cli, but behind the scenes it is doing a copy then delete for you - so you can make the operation easier, but it is not a true 'rename'.

https://docs.aws.amazon.com/cli/latest/reference/s3/mv.html

Upvotes: 6

Related Questions