CODEWITHSUNDEEP
google-bigquery
Mangai
Mangai

Reputation: 95

Max number of partitions that can be created on bigquery table

Im getting below error while moving data from unpartitioned to partitioned table.

"Resources exceeded during query execution: Table mydataset.mytable$20101213 will have 2501 partitions when the job finishes, exceeding limit 2500..'}]"

--> Whether bigquery limits on the max number of partitions that can be created on a table ? or whether this is just a day quota ?

Moreover, this limit is not mentioned in their BigQuery "Quotas and Limits" pages

Upvotes: 2

Views: 10502

Answers (3)

Graham Polley
Graham Polley

Reputation: 14781

It's mentioned in the docs. Currently, you cannot have more than 10,000 partitions per table. You'll need to split your data/partitions across multiple tables (each of around 7 years of data).

Each partitioned table can have up to 10,000 partitions.

https://cloud.google.com/bigquery/quotas#partitioned_tables

Upvotes: 10

Pete Heyworth
Pete Heyworth

Reputation: 31

As of 2024, the maximum has increased from 4000 to 10000 partitions per table.

https://cloud.google.com/bigquery/quotas#partitioned_tables

Replying since this page is high on the Google search results for "bigquery maximum partitions"

Upvotes: 2

Matt Laz
Matt Laz

Reputation: 230

Once your data is split across multiple tables you can then write a view to union the two tables together. Whenever the view is queried with the partitioned field in the where clause only the relevant partitions will be processed by BigQuery.

This results in a view with 4,000+ partitions which business users can query without worrying about which version of a table they need to use or union-ing the tables themselves.

Upvotes: 0

Related Questions