Reputation: 1
I`m facing an issue during ingestion process of a BigQuery table which contains RECORD (STRUCT) fields.
The Apache Beam (I`m using Scio with Scala) load process extracts a JSON file with a flat values for the RECORD fields. As shown below:
"organization_fields":"{org_status_acordo=false, org_inadimp=false, org_status_averb=false, cnpj_do_cadastro=}"
While exporting the same record directly from BigQuery, the JSON file contains:
"organization_fields":{ "org_status_acordo":"false", "org_inadimp":"false", "org_status_averb":"false", "cnpj_do_cadastro":"" },
I think this is the cause of throwing error:
"Error while reading data, error message: JSON parsing error in row starting at position 0: Flat value specified for record field"
Is there any limitation in using Apache Beam (Scio) ingestion BigQuery tables with RECORD data type?
Thanks for helping.
Find out if it`s possible to ingest BigQuery tables with RECORD data type using Apache Beam (Scio Distribution)
Upvotes: 0
Views: 170