CODEWITHSUNDEEP
google-bigquerygoogle-cloud-dataflowapache-beam
Sagi Mann
Sagi Mann

Reputation: 3610

Google Dataflow: how to insert RECORD non-repeated type field to Big Query?

I'm new to Dataflow. I've got a predefined-schema containing a non-repeated RECORD field called "device":

device.configId: STRING
device.version: STRING

Using a ParDo transform, I tried inserting a TableRow with this kind of field, as follows:

TableRow row = new TableRow();
row.put("field1", "val1");
TableRow device = new TableRow();
device.put("configId", "conf1");
device.put("version", "1.2.3");
row.put("device", device);
out.output(row);

I logged the table row, it looks like this:

{field1=val1, device={configId=conf1, version=1.2.3}}

I output it to a standard transform: BigQueryIO.write()

But the latter issues an error:

java.lang.RuntimeException: java.io.IOException:
Insert failed: [{"errors":[{
    "debugInfo":"",
    "location":"device.configid",
    "message":"This field is not a record.",
    "reason":"invalid"
}],"index":0}]

Not sure why, but note the location spells "configid" in lowecase - not in camel case as in the original log.

Any ideas on how to insert such an object to BigQuery?

Upvotes: 1

Views: 1089

Answers (1)

Sagi Mann
Sagi Mann

Reputation: 3610

Found out the problem. Apparently, this error message was caused only when the "configId" field was set to null rather than "conf1". To be exact, it was implicitly set to JSONObject.NULL coming from some input object.

Upvotes: 2

Related Questions