CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-storage
kakaji
kakaji

Reputation: 191

S3 to Google Storage using transfer service

In AWS I have folder format like eg : Bucketname/Data/files/abc_01-02-2022.csv In a increment order I have files for each dates for all the months in year.

In Google Cloud Storage I am trying to create folder structure like eg:Bucketname/data/202202/files/abc_01-02-2022.csv for whole year

So, I am trying to use storage transfer service which will take dynamically or from object itself and create a folder structure automatically by getting trigger automatically 2nd of the month.

Can we achieve this by using transfer service. what is the best way to achieve this I am trying to make it simple as possible

Upvotes: 1

Views: 668

Answers (2)

Ajitesh Abhishek
Ajitesh Abhishek

Reputation: 21

This is possible via using STS API. You can specify "path" at the destination bucket.

Upvotes: 0

Zeenath S N
Zeenath S N

Reputation: 1170

Storage Transfer Service does not support destination object prefixes, the reason behind it is, Storage Transfer Service doesn’t support remapping, that is, you cannot copy the path Bucketname/Data/files/ to Bucketname/data/202202/files

My recommendation would be to first use the Storage Transfer Service to copy everything from one bucket to another and later use any of the available methods to rename the object in the new bucket to Bucketname/data/202202/files.

Also the Cloud Storage Objects are flat namespaces, that is, Cloud Storage does not have folders and sub folders. There are a few documents that you can refer to for more information on this Object name considerations and Folders

Upvotes: 1

Related Questions