CODEWITHSUNDEEP
mongodbgoogle-cloud-platformgoogle-cloud-datastore
Soumithri Chilakamarri
Soumithri Chilakamarri

Reputation: 620

Transfer MongoDB dump on external hard drive to google cloud platform

As a part of my thesis project, I have been given a MongoDB dump of size 240GB which is on my external hard drive. I'll have to use this data to run my python scripts for a short duration. However, since my dataset is huge and I cannot mongoimport on my local mongodb server (since I don't have enough internal memory), my professor gave me a $100 google cloud platform coupon so I can use the google cloud computing resources.

So far I have researched that I can do it this way:

Create a compute engine in GCP and install mongodb on remote engine. Transfer the MongoDB dump to remote instance and run the scripts to get the output.

This method works well but I'm looking for a method to create a remote database server in GCP so I that I can run my scripts locally, which is something like one of the following.

  1. Creating a remote mongodb server on GCP so that I can establish a remote mongo connection to run my scripts locally.

  2. Transferring the mongodb dump to google's datastore so then I can use the datastore API to remotely connect and run my scripts locally.

I have given a thought of using MongoDB atlas but because of the size of the data, I will be billed hugely and I cannot use my GCP coupon.

Any help or suggestions on how of either of the two methods can be implemented is appreciated.

Upvotes: 0

Views: 1197

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75910

There is 2 parts to your question

First, you can create a compute engine VM with MongoDB installed and load your backup on it. Then, open the right firewall rules for allowing the connexion from your local environment to the Google Compute Engine VM. The connexion will be performed with a simple login/password.

You can use a static IP on your VM. By the way, in case of reboot on the VM you will keep the same IP (and it will be easier for your local connexion).

Second, BE CAREFUL to datastore. It's a good product, serverless NoSQL database, document oriented, but it's absolutely not the MongoDB equivalent. You can't perform aggregate, you are limited in search capabilities,... It's designed for specific use case (I don't know yours, but don't think that is the MongoDB equivalent!).

Anyway, if you use Datastore, you will have to use a service account or to install Google Cloud SDK on your local environment to be authenticated and to be able to request Datastore API. No login/password in this case.

Upvotes: 2

Related Questions