CODEWITHSUNDEEP
amazon-s3google-cloud-storage
Dipu Muraleedharan
Dipu Muraleedharan

Reputation: 43

How to Transfer data from Google cloud storage to S3 without any manual activity?

I want to transfer data from my cloud storage bucket to S3. I did a lot of research . I found an interesting article in which this can be done using gsutil command. Again it requires a manual activity. Is there any way to do this without any manual activity or like any possibility to run this gsutil command through Java API ?

I have created a VM instance on my Google cloud project.I have configured the boto file which has my AWS credentials.Using gsutil command I was able to copy my data from cloud storage to s3.I want to automate this step

Upvotes: 2

Views: 5632

Answers (3)

swooders
swooders

Reputation: 189

Skyplane is a much faster alternative for transferring data between clouds (up to 110x for large files). You can transfer data with the command:

skyplane cp -r s3://aws-bucket-name/ gcs://google-bucket-name/

Upvotes: 0

Anton
Anton

Reputation: 4052

Fully automated and easy to use way would be Google Storage Transfer. It would let you select the source and target buckets in either S3 or GCS and set shedule such as "run daily at 1am"

Upvotes: 0

Anton
Anton

Reputation: 4052

gsutil can work with both Google Storage and S3.

gsutil rsync -d -r gs://my-gs-bucket s3://my-s3-bucket

You just need to configure it with both - Google and your AWS S3 credentials. gsutil would use credentials from ~/.boto file.

There are multiple ways to automate shell command execution. From Java's Runtime.exec() to Google Cloud Scheduler (your cloud's cron)

Upvotes: 1

Related Questions