CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
GabrielBoehme
GabrielBoehme

Reputation: 322

BigQuery errors with rateLimitExceeded even not having 100 concurrent jobs

Over the past few days, many of the requests made to Big Query are failing with:

Exceeded rate limits: too many concurrent queries for this project_and_region. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

When I go to the documentation, I get that this error is related to running too many concurrent jobs in Big Query (the limit is 100). When I go to Big Query admin -> monitoring and check the job concurrency metric, at no point in time there were more than 50 concurrent tasks running (seeing the chart for the past few hours with 5-minute intervals), even though there were failures in that meantime.

I'm also checking the tasks that failed thru the INFORMATION_SCHEMA.JOBS table, for getting accurate timestamps on the failed tasks, and checking the concurrency for that timestamps on the admin monitoring.

We're not using queues in our environment for now.

What am I missing?

Upvotes: 0

Views: 811

Answers (1)

KarlVKVA
KarlVKVA

Reputation: 58

One of the reasons for this error is the slots reservation already maxed out while the tasks were running, there are quotas applied to reservations per region.

For EU maximum number of slots that can be purchased - 2000 slots

For US maximum number of slots that can be purchased - 4000 slots

You can check the chart of the reservation or slot usage and see if it maxes out while the concurrent tasks are running. Even if the concurrent tasks did not exceed 100 but the reservation is maxed out during the process, you will encounter this error.

You can learn more about the quotas of reservations per region using the link below:

https://cloud.google.com/bigquery/quotas#reservations

If you think you did not exceed this limit, you can also reach out to google cloud support, so they can assist you on this problem.

https://cloud.google.com/support

Upvotes: 3

Related Questions