CODEWITHSUNDEEP
google-bigquery
Scicrazed
Scicrazed

Reputation: 602

BigQuery Count Appears to be Processing Data

I noticed that running a SELECT count(*) FROM myTable on my larger BQ tables yields long running times, upwards of 30/40 seconds despite the validator claiming the query processes 0 bytes. This doesn't seem quite right when 500 GB queries run faster. Additionally, total row counts are listed under details -> Table Info. Am I doing something wrong? Is there a way to get total row counts instantly?

Upvotes: 0

Views: 487

Answers (1)

Tamir Klein
Tamir Klein

Reputation: 3632

When you run a count BigQuery still needs to allocate resources (such as: slot units, shards etc). You might be reaching some limits which cause a delay. For example, the slots default per project is 2,000 units.

BigQuery execution plan provides very detail information about the process which can help you better understand the source of the delay. One way to overcome this is to use an approximate method described in this link

This Slide by Google might also help you

enter image description here

For more details see this video about how to understand the execution plan

Upvotes: 2

Related Questions