CODEWITHSUNDEEP
google-bigquery
Michael
Michael

Reputation: 1728

Load a nullable repeated field in BigQuery

I have a JSON payload like this that I’d like to stream insert into a table I’m managing:

{ “Field1”: “value1”, “Field2”: [“value1”, “value2”], }

Sometimes field2 is not present at all in the JSON. I thought repeated fields with type “STRING” could be null but I’m getting insert errors from the API saying it cannot be empty.

I’m doing the loading from python so can transform if necessary, I’m just not sure the best way to set the schema and JSON payload.

Upvotes: 3

Views: 3043

Answers (1)

Daniel Duato
Daniel Duato

Reputation: 378

I think I have been able to reproduce your issue using the method insert_rows, I’m going to assume that that is the method you used. The error returned by that method was [1]. Please let me know if you are accessing the API in a different way or getting a different error.

There seems to be an issue in that method of the library, since REPEATED fields are not being treated the way we would expect.

I have filled an issue regarding this problem [2].

As a workaround I recommend using the method insert_rows_json [3]. This method has the same functionality as insert_rows but not its issues.

[1] {'index': 0, 'errors': [{u'debugInfo': u'', u'reason': u'invalid', u'message': u'Field value cannot be empty.', u'location': u'field2'}]}

[2] https://github.com/googleapis/google-cloud-python/issues/9602

[3] https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client.insert_rows_json

Upvotes: 1

Related Questions