CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storagegsutilcp
Yanki Twizzy
Yanki Twizzy

Reputation: 8001

Copy files from multiple folders to corresponding folders within the same gcp bucket folder

I have a gcp bucket say gs://my-folder that contains folders that either start with newVersion or don't. Each of these folders contain files. For example I have gs://my-folder/newVersion=1, gs://my-folder/newVersion=2, gs://my-folder/newVersion=3. I want to copy whatever is in these folders into gs://my-folder/1, gs://my-folder/2 and gs://my-folder/3 respectively. Is there a way of using gsutil cp command to do this? It would have much been easier if the these were different folders.

I don't want to copy folders individually. I want a script maybe that could identify all the newVersion= folders and copy them to the corresponding folders.

Upvotes: 0

Views: 3034

Answers (2)

Catherine O
Catherine O

Reputation: 1011

You can simply use the gsutil cp command if you want to copy a folder and its content.

gsutil cp gs://my-folder/newVersion=1/* gs://my-folder/1

To copy the entire directory tree, you can use -r command.

gsutil cp -r gs://my-folder/newVersion=1/* gs://my-folder/1

However, note that this will only work if the folder to copy contains files. Since it expects actual objects/files to be copied and not empty directories.

On the other hand, using gsutil mv command will allow you to perform a copy from source to destination followed by removing the source for each object. If you don't want your original file/object to be removed, you can use the cp instead of mv.

Upvotes: 1

TheRealJamesKirk
TheRealJamesKirk

Reputation: 71

If it doesn't have to be a copy then just:

Renaming Groups Of Objects You can use the gsutil mv command to rename all objects with a given prefix to have a new prefix. For example, the following command renames all objects under gs://my_bucket/oldprefix to be under gs://my_bucket/newprefix, otherwise preserving the naming structure:

gsutil mv gs://my_bucket/oldprefix gs://my_bucket/newprefix Note that when using mv to rename groups of objects with a common prefix, you cannot specify the source URL using wildcards; you must spell out the complete name.

If you do a rename as specified above and you want to preserve ACLs, you should use the -p option (see OPTIONS).

If you have a large number of files to move you might want to use the gsutil -m option, to perform a multi-threaded/multi-processing move:

gsutil -m mv gs://my_bucket/oldprefix gs://my_bucket/newprefix

ref: https://cloud.google.com/storage/docs/gsutil/commands/mv

Upvotes: 1

Related Questions