CODEWITHSUNDEEP
hadoophdfscloudgoogle-cloud-dataproc
James
James

Reputation: 2331

How can I securely transfer my data from on-prem HDFS to Google Cloud Storage?

I have a bunch of data in an on-prem HDFS installation. I want to move some of it to Google Cloud (Cloud Storage) but I have a few concerns:

What is the best way to move data securely from my HDFS store to Cloud Storage?

Upvotes: 0

Views: 629

Answers (1)

James
James

Reputation: 2331

To move Data from an on-premise Hadoop cluster to Google Cloud Storage, you should probably use the Google Cloud Storage connector for Hadoop. You can install the connector in any cluster by following the install directions. As a note, Google Cloud Dataproc clusters have the connector installed by default.

Once the connector is installed, you can use DistCp to move the data from your HDFS to Cloud Storage. This will transfer data over the (public) internet unless you have a special interlink setup with Google Cloud. To this end, you can use a squid proxy and configure the Cloud Storage connector to use it.

Upvotes: 2

Related Questions