CODEWITHSUNDEEP
google-bigquery
shantanuo
shantanuo

Reputation: 32316

response too large in big query

I am not able to run the group by having query.

select phone_number, count(*) as cnt from Jan1013.abcd4 group by phone_number having cnt > 1

I get an error that says...

Error: Response too large to return.

Upvotes: 0

Views: 1126

Answers (1)

Jeremy Condit
Jeremy Condit

Reputation: 7046

BigQuery distributes the work of each query across many different nodes. The "Response too large" error occurs if the size of the response at any one node exceeds some threshold. If you try to perform a GROUP BY on a field with too many distinct values, then each individual node will need to compute a huge list of response rows and will eventually fail with this error.

One way to address the problem is to use GROUP EACH BY, which assigns records to nodes by the value of the group key, reducing the number of distinct values that each node needs to consider.

Another approach is to tweak your query so that you consider fewer distinct values, perhaps by adding a WHERE clause that selects a subset of the phone_number values.

See also: Filtering a query causes "Response too large" error

Upvotes: 4

Related Questions