Bruno Faleiros
Bruno Faleiros

Reputation: 1

Scio (Apache Beam) BigQuery struct (Record) type issue

I`m facing an issue during ingestion process of a BigQuery table which contains RECORD (STRUCT) fields.

The Apache Beam (I`m using Scio with Scala) load process extracts a JSON file with a flat values for the RECORD fields. As shown below:

"organization_fields":"{org_status_acordo=false, org_inadimp=false, org_status_averb=false, cnpj_do_cadastro=}"

While exporting the same record directly from BigQuery, the JSON file contains: "organization_fields":{ "org_status_acordo":"false", "org_inadimp":"false", "org_status_averb":"false", "cnpj_do_cadastro":"" },

I think this is the cause of throwing error:

"Error while reading data, error message: JSON parsing error in row starting at position 0: Flat value specified for record field"

Is there any limitation in using Apache Beam (Scio) ingestion BigQuery tables with RECORD data type?

Thanks for helping.

Find out if it`s possible to ingest BigQuery tables with RECORD data type using Apache Beam (Scio Distribution)

Upvotes: 0

Views: 170

Answers (0)

Related Questions