CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-storage
kapil dev
kapil dev

Reputation: 111

Google cloud function to copy all data of source bucket to another bucket using python

I want to copy data from one bucket to another bucket using google cloud function. At this time I am able to copy only a single file to destination but I want to copy all files, folders, and sub-folders to my destination bucket.


from google.cloud import storage
def copy_blob(bucket_name= "loggingforproject", blob_name= "assestnbfile.json", destination_bucket_name= "test-assest", destination_blob_name= "logs"):
    """Copies a blob from one bucket to another with a new name."""
    bucket_name = "loggingforproject"
    blob_name = "assestnbfile.json"
    destination_bucket_name = "test-assest"
    destination_blob_name = "logs"

    storage_client = storage.Client()

    source_bucket = storage_client.bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.bucket(destination_bucket_name)

    blob_copy = source_bucket.copy_blob(
        source_blob, destination_bucket, destination_blob_name
    )

    print(
        "Blob {} in bucket {} copied to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    )

Upvotes: 5

Views: 7709

Answers (3)

Victor Nava
Victor Nava

Reputation: 1

Here is my typescript code, I call it from my website when a need to move images.

    exports.copiarImagen = functions.https.onCall(async (data, response) => {   
    var origen = data.Origen;
    var destino = data.Destino;
    console.log('Files:');
    const [files] = await admin.storage().bucket("bucket´s path").getFiles({ prefix: 'path where your images are"});
    files.forEach(async file => {      
      var nuevaRuta = file.name;
      await admin.storage().bucket("posavka.appspot.com").file(file.name)
      .copy(admin.storage().bucket("posavka.appspot.com").file(nuevaRuta.replace(origen,destino)));
      await admin.storage().bucket("posavka.appspot.com").file(file.name).delete();
    });

First I get all files in a specific path, then I copy those files to the new path, and finally I delete the file on the old path

I hope it helps you :D

Upvotes: 0

Deniss T.
Deniss T.

Reputation: 2652

Using gsutil cp is a good option. However, if you want to copy the files using Cloud Functions - it can be achieved as well.

At the moment, your function only copies a single file. In order to copy the whole content of your bucket you would need to iterate through the files within it.

Here is a code sample that I wrote for an HTTP Cloud Function and tested - you can use it for a reference:

MAIN.PY

from google.cloud import storage

def copy_bucket_files(request):
    """
    Copies the files from a specified bucket into the selected one.
    """

    # Check if the bucket's name was specified in the request
    if request.args.get('bucket'):
        bucketName = request.args.get('bucket')
    else:
        return "The bucket name was not provided. Please try again."

    try:
        # Initiate Cloud Storage client
        storage_client = storage.Client()
        # Define the origin bucket
        origin = storage_client.bucket(bucketName)
        # Define the destination bucket
        destination = storage_client.bucket('<my-test-bucket>')

        # Get the list of the blobs located inside the bucket which files you want to copy
        blobs = storage_client.list_blobs(bucketName)

        for blob in blobs:
            origin.copy_blob(blob, destination)

        return "Done!"

    except:
        return "Failed!"

REQUIREMENTS.TXT

google-cloud-storage==1.22.0

How to call that function:

It can be called via the URL provided for triggering the function, by appending that URL with /?bucket=<name-of-the-bucket-to-copy> (name without <, >):

https://<function-region>-<project-name>.cloudfunctions.net/<function-name>/?bucket=<bucket-name>

Upvotes: 5

Dustin Ingram
Dustin Ingram

Reputation: 21580

You can use the gsutil cp command for this:

gsutil cp gs://first-bucket/* gs://second-bucket

See https://cloud.google.com/storage/docs/gsutil/commands/cp for more details

Upvotes: 2

Related Questions