CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storage
knoll
knoll

Reputation: 295

Google Cloud Storage Transfer Sink Prefix

I've scanned the Storage Transfer docs, but I'm not seeing an example of this scenario. For reference, I'm using this script - https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/transfer_service/nearline_request.py

Is there a way to set a prefix for files being transferred from the source bucket to the sink bucket?

For example, if I have this file in the source bucket:
gs://source_bucket/myFile.csv
I want to add a prefix to it so that it looks like this in the sink bucket:
gs://sink_bucket/date/type/myFile.csv

As far as I can tell, the sink bucket argument only accepts a bucket's name. I'm not seeing an option for something like this in the transferSpec docs either. Any help would be most appreciated.

Upvotes: 0

Views: 950

Answers (1)

Daniel Ocando
Daniel Ocando

Reputation: 3794

Notice that as stated on the documentation the Storage Transfer Service does not support destination object prefixes.The main reason behind it being that it currently does not support remapping.

My recommendation would be to first use the Storage Transfer service to copy everything from one bucket to another and only afterwards use any of the available methods to rename the object in the new bucket to /date/type/myFile.csv. As for Cloud Storage objects don't reside within subdirectories and they merely exist with the name /date/type/myFile.csv.

Upvotes: 2

Related Questions