CODEWITHSUNDEEP
pythonamazon-s3google-cloud-storageboto
martinredolatti
martinredolatti

Reputation: 48

Copy files from S3 to Google Cloud storage using boto

I'm trying to automate copying of files from S3 to Google Cloud Storage inside a python script.

Everywhere I look people recommend using gsutil as a command line utility.

Does anybody know if this copies files directly? Or does it first download the files to the computer and then uploads them to GS?

Can this be done using the boto library and google's OAuth2 plugin?

This is what i've got from google's documentation and a little bit or trial-error:

src_uri = boto.storage_uri('bucket/source_file')          
dst_uri = boto.storage_uri('bucket/destination_file', 'gs')                

object_contents = StringIO.StringIO()                                        

src_uri.get_key().get_file(object_contents)                                  

object_contents.seek(0)                                                      

dst_uri.set_contents_from_file(object_contents)                              

object_contents.close()

From what I understand I'm reading from a file into an object in the host where the script is running, and later uploading such content into a file in GS.

Is this right?

Upvotes: 2

Views: 4573

Answers (1)

David
David

Reputation: 9731

Since this question was asked, GCS Transfer Service has become available. If you want to copy files from S3 to GCS without an intermediate, this is a great option.

Upvotes: 1

Related Questions