CODEWITHSUNDEEP
google-bigquery
ANSHUMAN DWIVEDI
ANSHUMAN DWIVEDI

Reputation: 11

"Exceeded rate limits: exceeded quota for Streaming Table Creation Rate in bigquery

while streaming inserts to bq I got this quota error which i couldn't find anywhere in official documentation.

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Exceeded rate limits: Your Project: **** exceeded quota for Streaming Table Creation Rate. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Exceeded rate limits: Your Project: **** exceeded quota for Streaming Table Creation Rate. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors",
  "status" : "PERMISSION_DENIED"
}

Upvotes: 1

Views: 379

Answers (1)

Whit Waldo
Whit Waldo

Reputation: 5217

If you follow the URL in your error reason to a list of the error messages you might receive, you'll see that you've been rate limited.

See the rate limits for query jobs here and rate limits for API requests here.

Update

Google doesn't appear to differentiate by streaming or not, but they do cite table quotas at 1500 DML operations per table per day (not what you were doing, but still useful to note) and a max of 5 DDL operations every 10 seconds (and creating tables would fall under this bucket).

Limit your new table creations to 5 every 10 seconds or so and you shouldn't run afoul of the quotas.

Upvotes: 2

Related Questions