CODEWITHSUNDEEP
google-bigquery
Kenta Kozuka
Kenta Kozuka

Reputation: 47

BigQuery: --[no]use_avro_logical_types flag doesn't work

I try to use bq command with --[no]use_avro_logical_types flag to load avro files into BigQuery table which does not exist before executing the command. The avro schema contains timestamp-millis logical type value. When the command is executed, a new table is created but the schema of its column becomes INTEGER. This is a recently released feature so that I cannot find examples and I don't know what I am missing. Could anyone give me a good example?

My avro schema looks like following,

...
  }, {
    "name" : "timestamp",
    "type" : [ "null", "long" ],
    "default" : null,
    "logicalType" : [ "null", "timestamp-millis" ]
  }, {
...

And executing command is this: bq load --source_format=AVRO --use_avro_logical_types <table> <path/to/file>

Upvotes: 0

Views: 1368

Answers (2)

Kevin Liu
Kevin Liu

Reputation: 26

To use the timestamp-millis logical type, you can specify the field in the following way:

{
  "name" : "timestamp",
  "type" : {"type": "long", "logicalType" : "timestamp-millis"}
}

In order to provide an optional 'null' value, you can try out the following spec:

{
  "name" : "timestamp",
  "type" : ["null", {"type" : "long", "logicalType" : "timestamp-millis"}]
}

For a full list of supported Avro logical types please refer to the Avro spec: https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types.

Upvotes: 1

Lisa Yin
Lisa Yin

Reputation: 169

According to https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro, the avro type, timestamp-millis, is converted to an INTEGER once loaded in BigQuery.

Upvotes: 0

Related Questions