CODEWITHSUNDEEP
google-bigquery
krtek
krtek

Reputation: 1065

BigQuery: What does 'too many large queries' error mean?

We are getting following error in our Google BigQuery application:

Exceeded quota: too many large queries for this project

What does this mean? What is "large query"? Can we raise this quota somehow?

Upvotes: 2

Views: 323

Answers (1)

Jordan Tigani
Jordan Tigani

Reputation: 26617

BigQuery allows queries that are processing up to 400 GB of data to execute simultaneously, plus 2 queries of any size. It is this latter quota that you are hitting.

We call the 400 GB capacity the 'pipe'. You can keep that pipe full with up to 20 queries at once that add up to less than 400GB. Once you go outside that pipe, you can run two additional queries of any size.

If this sounds confusing, let's say you have two queries that read 150 GB, one that reads 1TB of data, and one that reads 500GB. The two smaller queries will fit in the 400GB pipe, but the 500GB and 1TB query will be considered 'large' queries. These should all be able to run together under the quota limits.

However, if the 150 GB queries had been 250GB, they wouldn't have both fit in the 400GB pipe, and one of them would have been considered a 'large' query, so you'd only be able to run either the 1TB or the 500GB query.

The BigQUery team is usually pretty reluctant to raise this particular quota in order to prevent customers from being able to degrade performance for other customers. If you have strong reasons for needing the quota bump, we may be able to accommodate you if you contact us via http://www.google.com/enterprise/cloud/contact.html.

Another option is to run some of your large queries at batch priority. Batch priority means it may take much longer for your results, but you are not subject to the same limits as interactive queries.

Upvotes: 2

Related Questions