CODEWITHSUNDEEP
google-bigquerylimitdbtquota
dlsupaolo11
dlsupaolo11

Reputation: 33

DBT BigQuery error: Quota exceeded: Your table exceeded quota for imports or query appends per table

We've got an instance of DBT running on BigQuery, and as the number of models have increased over time (more projects, more stakeholders, more data to cover), increasingly from time to time we've been getting DBT job failures with the BigQuery error: Quota exceeded: Your table exceeded quota for imports or query appends per table.

Normally we ignore it and just let our jobs run successfully again the next time around (or manually trigger them again if urgent). But it really sucks when by chance this happens during a scheduled job run at a critical time of day (or when everyone's asleep).

The error message isn't very specific with what exact quota or limit was breached:

But the closest one seems to be this one (though I somehow doubt our DBT triggers 1500 query jobs to a specific table in a day): enter image description here

It says that its a limit that cannot be increased, nor can I find it under the Quotas & System Limits page in the Google Cloud Console.

Could I be mistaken? And if it cannot be changed, any advice on how to mitigate the chances of hitting this limit, especially with a DBT/BigQuery environment that's only getting bigger with our business needs and usage?

Upvotes: 0

Views: 88

Answers (0)

Related Questions