CODEWITHSUNDEEP
google-bigquerygoogle-cloud-datastore
David Cesar Santos
David Cesar Santos

Reputation: 449

Export Google Cloud Datastore and import to BigQuery programmatically

I'm looking for a method to export my Cloud Datastore and import it into BigQuery daily. The manual way is described at google page. I do not find a clean way to automate it.

Upvotes: 4

Views: 5041

Answers (4)

Christiaan
Christiaan

Reputation: 2725

As of last week there's a proper way to automate this. The most important part is gcloud beta datastore export.

I created a script around it: https://github.com/chees/datastore2bigquery You could run this in a cron job.

See here for a demo of how it works: https://www.youtube.com/watch?v=dGyQCE3bWkU

Upvotes: 4

Meh Nori
Meh Nori

Reputation: 11

It is possible using the following code. It basically uses App Engine Cron jobs and BigQuery API.

https://github.com/wenzhe/appengine_datastore_bigquery

Upvotes: -1

Tim D
Tim D

Reputation: 205

Building on @Jordan's answer above, the steps to do this would be:

1) Make a storage bucket

2) Export datastore entities to this bucket

3) Open Big Query Web UI, and load using the Google Cloud file path.

Full tutorial with images is available at this post.

Upvotes: 1

Jordan Tigani
Jordan Tigani

Reputation: 26637

There isn't a simple way to do this, but you can separate out the two parts: creating appengine backups and loading them into bigquery.

You can use scheduled backups to create datastore backups periodically (https://cloud.google.com/appengine/articles/scheduled_backups).

You can then use Apps Script to automate the BigQuery portion (https://developers.google.com/apps-script/advanced/bigquery#load_csv_data) or use an AppEngine cron to do the same thing.

Upvotes: 5

Related Questions