CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
David542
David542

Reputation: 110502

Most efficient way to retrieve ~100MB query result size in BigQuery

Using jobs.query has the limitation of 10MB byteSize, and so I have to pagination/request 10 times in order to retrieve 100MB data in BigQuery.

I am looking for how to run a query and download about the result (~100MB) in the fastest possible time. What would be the suggested way to do this?

https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query

Additionally, is there a way to retrieve the query result in a compressed format (if that would speed things up at all?)

Upvotes: 0

Views: 640

Answers (1)

Felipe Hoffa
Felipe Hoffa

Reputation: 59325

Either:

  1. Run an export job. Specify gzip compression. Load the file from GCS.

  2. Use the new BigQuery Storage API.

I created a feature request you can upvote to get a CLI tool to download with the Storage API:

Upvotes: 2

Related Questions