CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beam
Mohammed Niaz
Mohammed Niaz

Reputation: 396

Parse JSON as key value in Dataflow job

How to parse JSON data in apache beam and store in bigquery table ? For example: JSON data

[{ "name":"stack"},{"id":"100"}].

How to parse JSON data and convert to PCollection K,V that will store in BQ table? Appreciate your help!!

Upvotes: 0

Views: 2030

Answers (1)

Alex Amato
Alex Amato

Reputation: 1591

Typically you would use a built in JSON parser in the programming language (Are you using beam or python). Then create a TableRow object and use that for the PCollection which you are passing to the BQ table.

Note: Some JSON parsers disallow JSON which starts with a root list, as you have shown in your example. They tend to prefer something like this, with a root map. I believe this is the case in python's json library.

{"name":"stack", "id":"100"}

Please see this example pipeline, for an example on how to create the PCollection and use BigqueryIO.

You may also want to consider using one of the X to BigQuery template pipelines.

Upvotes: 2

Related Questions