CODEWITHSUNDEEP
google-bigquery
Zizon
Zizon

Reputation: 41

Native JSON support for BigQuery?

Is there any plan for Google BigQuery to implement native JSON support?

I am considering migrating hive data (~20T) to Google BigQuery, but the table definitions in Hive contains map type which is not supported in BigQuery.

for example, the HiveQL below:

select gid, payload['src'] from data_repository;

although, it can be worked around by using regular expression.

Upvotes: 4

Views: 1201

Answers (2)

Michael Sheldon
Michael Sheldon

Reputation: 2057

As of 1 Oct 2012, BigQuery supports newline separated JSON for import and export.

Blog post: http://googledevelopers.blogspot.com/2012/10/got-big-json-bigquery-expands-data.html

Documentation on data formats: https://developers.google.com/bigquery/docs/import#dataformats

Upvotes: 3

Jordan Tigani
Jordan Tigani

Reputation: 26637

Your best bet is to coerce all of your types into csv before importing, and if you have complex fields, decompose them via a regular expression in the query (as you suggested).

That said, we are actively investigating support for new input formats, and are interested in feedback as to what formats would be the most useful. There is support in the underlying query engine (Dremel) for types similar to the hive map type, but BigQuery, however, does not currently expose a mechanism for ingesting nested records.

Upvotes: 0

Related Questions