CODEWITHSUNDEEP
google-cloud-dataflow
Kepper
Kepper

Reputation: 13

How to create dataflow pipeline from JSON file on cloud storage to datastore

Im new to GCP and needed some help on the following: I have a .json file uploaded to cloud storage and need to move the data into cloud datastore for parsing/queries.

I think a large dataset may take too long to import natively, so was interesting in using dataflow to transform and load. Any ideas or help would be much appreciated.

Upvotes: 1

Views: 3209

Answers (1)

Yuri Grinshteyn
Yuri Grinshteyn

Reputation: 727

This is a fairly straightforward problem. You'll need to:

  1. Review the basics of writing dataflow pipelines here: https://beam.apache.org/documentation/pipelines/design-your-pipeline/

  2. Read from GCS: https://beam.apache.org/documentation/sdks/javadoc/0.2.0-incubating/org/apache/beam/sdk/io/TextIO.html

  3. Transform JSON to entities: https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/coders/TableRowJsonCoder (or similar)

  4. Write to Datastore https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore

Hope this helps!

Upvotes: 3

Related Questions