CODEWITHSUNDEEP
google-bigquerysizelimit
Shivam Tripathi
Shivam Tripathi

Reputation: 445

Google BigQuery set limit on query size using estimates

Google BigQuery has a dry-run feature by which we can get an estimate of the total query size before running it.
Is there some hack/mechanism on the BigQuery by which we could set up an upper limit on the allowed size of queries, and automatically reject any queries that exceed that size?

This would be useful, so as to avoid making any queries accidentally that might possibly exceed the set limit (on WEB UI, COMMAND-LINE and API).

Upvotes: 3

Views: 4851

Answers (1)

Pentium10
Pentium10

Reputation: 207982

You can limit the number of bytes billed for a query using the maximum bytes billed setting. When you set maximum bytes billed, if the query will read bytes beyond the limit, the query fails without incurring a charge.

If a query fails because of the maximum bytes billed setting, an error like the following is returned:

Error: Query exceeded limit for bytes billed: 1000000. 10485760 or higher required.

In the BigQuery web UI, enter an integer in the Maximum Bytes Billed field in the query options.

enter image description here

In the CLI, use bq query command with the --maximum_bytes_billed flag.

bq --location=US query --maximum_bytes_billed=1000000 --use_legacy_sql=false "SELECT word FROM bigquery-public-data.samples.shakespeare"

In the API, set the maximumBytesBilled property.

See more in BigQuery Best Practices: Controlling Costs

Upvotes: 7

Related Questions