CODEWITHSUNDEEP
google-bigquerygoogle-cloud-datastore
thstart
thstart

Reputation: 427

How to export from BigQuery to Datastore?

I have tables in BigQuery which I want to export and import in Datastore. How to achieve that?

Upvotes: 2

Views: 2408

Answers (3)

Michel Hua
Michel Hua

Reputation: 1777

2020 anwer, use GoogleCloudPlatform/DataflowTemplates, BigQueryToDatastore

# Builds the Java project and uploads an artifact to GCS
mvn compile exec:java \
-Dexec.mainClass=com.google.cloud.teleport.templates.BigQueryToDatastore \
-Dexec.cleanupDaemonThreads=false \
-Dexec.args=" \
--project=<project-id> \
--region=<region-name> \
--stagingLocation=gs://<bucket-name>/staging \
--tempLocation=gs://<bucket-name>/temp \
--templateLocation=gs://<bucket-name>/templates/<template-name>.json \
--runner=DataflowRunner"

# Uses the GCS artifact to run the transfer job
gcloud dataflow jobs run <job-name> \
--gcs-location=<template-location> \
--zone=<zone> \
--parameters "\
readQuery=SELECT * FROM <dataset>.<table>,readIdColumn=<id>,\
invalidOutputPath=gs://your-bucket/path/to/error.txt,\
datastoreWriteProjectId=<project-id>,\
datastoreWriteNamespace=<namespace>,\
datastoreWriteEntityKind=<kind>,\
errorWritePath=gs://your-bucket/path/to/errors.txt"

I hope this will get a proper user interface in GCP Console on day! (as this is already possible for Pub/Sub to BigQuery using Dataflow SQL)

Upvotes: 1

Pravanjan
Pravanjan

Reputation: 1016

Table from BigQuery can be exported and imported to your datastore.

Download the jar file from https://github.com/yu-iskw/bigquery-to-datastore/releases

Then run the command 

java -cp bigquery-to-datastore-bundled-0.5.1.jar com.github.yuiskw.beam.BigQuery2Datastore --project=yourprojectId --runner=DataflowRunner  --inputBigQueryDataset=datastore  --inputBigQueryTable=metainfo_internal_2 --outputDatastoreNamespace=default --outputDatastoreKind=meta_internal  --keyColumn=key --indexedColumns=column1,column2 --tempLocation=gs://gsheetbackup_live/temp  --gcpTempLocation=gs://gsheetlogfile_live/temp

--tempLocation and --gcpTempLocation are valid cloud storage bucket urls.

--keyColumn=key - the key here is the unique field on your big query table

Upvotes: 2

Jinjun
Jinjun

Reputation: 391

You may export BigQuery data to CSV, then import CSV into Datastore. The first step is easy and well documented https://cloud.google.com/bigquery/docs/exporting-data#exporting_data_stored_in_bigquery. For the second step, there are many resources that help you achieve that. For example,
https://groups.google.com/forum/#!topic/google-appengine/L64wByP7GAY
Import CSV into google cloud datastore

Upvotes: -1

Related Questions