CODEWITHSUNDEEP
google-bigqueryavro
Tobi
Tobi

Reputation: 936

How to export AVRO files from a BigQuery table with a DATE column and load it again to BigQuery

For moving data from a BigQuery (BQ) table that resides in the US, I want to export the table to a Cloud Storage (GCS) bucket in the US, copy it to an EU bucket, and from there import it again.

The problem is that AVRO does not support DATE types, but it is crucial to us as we are using the new partitioning feature that is not relying on ingestion time, but a column in the table itself.

The AVRO files contain the DATE column as a STRING and therefore a Field date has changed type from DATE to STRING error is thrown, when trying to load the files via bq load.

There has been a similar question, but it is about timestamps - in my case it absolutely needs to be a DATE as dates don't carry timezone information and timestamps are always interpreted in UTC by BQ.

It works when using NEWLINE_DELIMITED_JSON, but is it possible to make this work with AVRO files?

Upvotes: 1

Views: 1355

Answers (1)

Guillermo Cacheda
Guillermo Cacheda

Reputation: 2232

As @ElliottBrossard pointed out in the comments, there's a public feature request regarding this where it's possible to sign up for the whitelist.

Upvotes: 1

Related Questions