CODEWITHSUNDEEP
google-cloud-storage
leaveme_alone
leaveme_alone

Reputation: 686

Copying directly from S3 to Google Cloud Storage

I can migrate data from Amazon AWS S3 to Azure using AWS SDK for Java and Azure SDk for Java. Now I want to do migrate data from Amazon AWS S3 to Google Cloud storage using Java.

Upvotes: 19

Views: 33572

Answers (4)

Gabriel Hodoroaga
Gabriel Hodoroaga

Reputation: 353

Create a new .boto file

[Credentials]
aws_access_key_id = ACCESS_KEY_ID
aws_secret_access_key = SECRET_ACCESS_KEY

and this command

BOTO_CONFIG=.boto gsutil -m cp s3://bucket-name/filename gs://bucket-name

or this

BOTO_CONFIG=.boto gsutil -m cp gs://bucket-name/filename s3://bucket-name

Upvotes: 2

jterrace
jterrace

Reputation: 67133

The gsutil command-line tool supports S3. After you've configured gsutil, you'll see this in your ~/.boto file:

# To add aws credentials ("s3://" URIs), edit and uncomment the
# following two lines:
#aws_access_key_id = 
#aws_secret_access_key =

Fill in the aws_access_key_id and aws_secret_access_key settings with your S3 credentials and uncomment the variables.

Once that's set up, copying from S3 to GCS is as easy as:

gsutil cp -R s3://bucketname gs://bucketname

If you have a lot of objects, run with the -m flag to perform the copy in parallel with multiple threads:

gsutil -m cp -R s3://bucketname gs://bucketname

Upvotes: 52

Justin Poehnelt
Justin Poehnelt

Reputation: 3469

Use the Google Cloud Storage transfer tool.

Google Cloud Storage Transfer Tool

Upvotes: 18

Anurag
Anurag

Reputation: 916

The answer suggested by jterrace (aws key and secret in .boto file) is correct and worked for me for many regions but not for some regions that need only AWS Signature Version 4. For instance while connecting to 'Mumbai' region I got this error:

BadRequestException: 400 InvalidRequest The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256

In order to overcome this problem (make gsutil use AWS Signature v4) I had to add following additional lines to ~/.boto file. These lines create a new section [s3] in the config file:

[s3]
host = s3.ap-south-1.amazonaws.com
use-sigv4 = True

Reference:
Interoperability support for AWS Signature Version 4
Gsutil cannot copy to s3 due to authentication

Upvotes: 5

Related Questions