CODEWITHSUNDEEP
javagoogle-bigqueryresources
Tr909
Tr909

Reputation: 31

BigQuery problem resources exceeded during query execution

I am using BigQuery and get the following error when trying to execute a somehow complex query:

Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex.

I am using a WITH clause to create a view table. This table is populated from back-end Java and has a lot of rows (~900 rows). Below is the full query from my repository class:

with weights as (
select 3 as weight, 3000973 as providerId
union all select 3 as weight, 3001278 as providerId
union all select 3 as weight, 3000945 as providerId
union all select 1 as weight, 3000365 as providerId
union all select 1 as weight, 3000364 as providerId
union all select 1 as weight, 3000367 as providerId
union all select 1 as weight, 3000366 as providerId
union all select 1 as weight, 3000361 as providerId
union all select 1 as weight, 3000360 as providerId
union all select 1 as weight, 3000363 as providerId
union all select 1 as weight, 3000362 as providerId
union all select 1 as weight, 3000357 as providerId
union all select 1 as weight, 3000356 as providerId
union all select 1 as weight, 3000359 as providerId
union all select 1 as weight, 3000358 as providerId
union all select 1 as weight, 3000353 as providerId
union all select 1 as weight, 3000352 as providerId
union all select 1 as weight, 3000355 as providerId
union all select 1 as weight, 3000354 as providerId
union all select 1 as weight, 3000381 as providerId
union all select 1 as weight, 3000380 as providerId
union all select 1 as weight, 3000383 as providerId
union all select 1 as weight, 3000382 as providerId
union all select 1 as weight, 3000377 as providerId
union all select 1 as weight, 3000376 as providerId
union all select 1 as weight, 3000379 as providerId
union all select 1 as weight, 3000378 as providerId
union all select 1 as weight, 3000373 as providerId
union all select 1 as weight, 3000372 as providerId
union all select 1 as weight, 3000375 as providerId
union all select 1 as weight, 3000374 as providerId
union all select 1 as weight, 3000369 as providerId
union all select 1 as weight, 3000368 as providerId
union all select 1 as weight, 3000371 as providerId
union all select 1 as weight, 3000370 as providerId
union all select 1 as weight, 3000333 as providerId
union all select 1 as weight, 3000332 as providerId
union all select 1 as weight, 3000335 as providerId
union all select 1 as weight, 3000334 as providerId
union all select 1 as weight, 3000329 as providerId
union all select 1 as weight, 3000328 as providerId
union all select 1 as weight, 3000330 as providerId
union all select 1 as weight, 3000325 as providerId
union all select 1 as weight, 3000324 as providerId
union all select 1 as weight, 3000326 as providerId
union all select 1 as weight, 3000321 as providerId
union all select 1 as weight, 3000320 as providerId
union all select 1 as weight, 3000323 as providerId
union all select 1 as weight, 3000322 as providerId
union all select 1 as weight, 3000349 as providerId
union all select 1 as weight, 3000348 as providerId
union all select 1 as weight, 3000351 as providerId
union all select 1 as weight, 3000350 as providerId
union all select 1 as weight, 3000345 as providerId
union all select 1 as weight, 3000344 as providerId
union all select 1 as weight, 3000347 as providerId
union all select 1 as weight, 3000346 as providerId
union all select 1 as weight, 3000341 as providerId
union all select 1 as weight, 3000340 as providerId
union all select 1 as weight, 3000343 as providerId
union all select 1 as weight, 3000342 as providerId
union all select 1 as weight, 3000337 as providerId
union all select 1 as weight, 3000336 as providerId
union all select 1 as weight, 3000339 as providerId
union all select 1 as weight, 3000338 as providerId
union all select 1 as weight, 3000429 as providerId
union all select 1 as weight, 3000428 as providerId
union all select 1 as weight, 3000431 as providerId
union all select 1 as weight, 3000430 as providerId
union all select 1 as weight, 3000425 as providerId
union all select 1 as weight, 3000424 as providerId
union all select 1 as weight, 3000427 as providerId
union all select 1 as weight, 3000426 as providerId
union all select 1 as weight, 3000422 as providerId
union all select 1 as weight, 3000417 as providerId
union all select 1 as weight, 3000416 as providerId
union all select 1 as weight, 3000419 as providerId
union all select 1 as weight, 3000445 as providerId
union all select 1 as weight, 3000444 as providerId
union all select 1 as weight, 3000447 as providerId
union all select 1 as weight, 3000446 as providerId
union all select 1 as weight, 3000441 as providerId
union all select 1 as weight, 3000440 as providerId
union all select 1 as weight, 3000443 as providerId
union all select 1 as weight, 3000442 as providerId
union all select 1 as weight, 3000437 as providerId
union all select 1 as weight, 3000436 as providerId
union all select 1 as weight, 3000439 as providerId
union all select 1 as weight, 3000438 as providerId
union all select 1 as weight, 3000433 as providerId
union all select 1 as weight, 3000432 as providerId
union all select 1 as weight, 3000435 as providerId
union all select 1 as weight, 3000434 as providerId
union all select 1 as weight, 3000396 as providerId
union all select 1 as weight, 3000399 as providerId
union all select 1 as weight, 3000398 as providerId
union all select 1 as weight, 3000393 as providerId
union all select 1 as weight, 3000392 as providerId
union all select 1 as weight, 3000395 as providerId
union all select 1 as weight, 3000389 as providerId
union all select 1 as weight, 3000388 as providerId
union all select 1 as weight, 3000391 as providerId
union all select 1 as weight, 3000390 as providerId
union all select 1 as weight, 3000384 as providerId
union all select 1 as weight, 3000387 as providerId
union all select 1 as weight, 3000413 as providerId
union all select 1 as weight, 3000412 as providerId
union all select 1 as weight, 3000415 as providerId
union all select 1 as weight, 3000414 as providerId
union all select 1 as weight, 3000411 as providerId
union all select 1 as weight, 3000410 as providerId
union all select 1 as weight, 3000405 as providerId
union all select 1 as weight, 3000407 as providerId
union all select 1 as weight, 3000406 as providerId
union all select 1 as weight, 3000401 as providerId
union all select 1 as weight, 3000400 as providerId
union all select 1 as weight, 3000403 as providerId
union all select 1 as weight, 3000402 as providerId
union all select 1 as weight, 3000493 as providerId
union all select 1 as weight, 3000492 as providerId
union all select 1 as weight, 3000495 as providerId
union all select 1 as weight, 3000494 as providerId
union all select 1 as weight, 3000489 as providerId
union all select 1 as weight, 3000488 as providerId
union all select 1 as weight, 3000491 as providerId
union all select 1 as weight, 3000490 as providerId
union all select 1 as weight, 3000485 as providerId
union all select 1 as weight, 3000484 as providerId
union all select 1 as weight, 3000487 as providerId
union all select 1 as weight, 3000486 as providerId
union all select 1 as weight, 3000481 as providerId
union all select 1 as weight, 3000480 as providerId
union all select 1 as weight, 3000483 as providerId
union all select 1 as weight, 3000482 as providerId
union all select 1 as weight, 3000509 as providerId
union all select 1 as weight, 3000508 as providerId
union all select 1 as weight, 3000511 as providerId
union all select 1 as weight, 3000510 as providerId
union all select 1 as weight, 3000504 as providerId
union all select 1 as weight, 3000507 as providerId
union all select 1 as weight, 3000506 as providerId
union all select 1 as weight, 3000501 as providerId
union all select 1 as weight, 3000500 as providerId
union all select 1 as weight, 3000503 as providerId
union all select 1 as weight, 3000502 as providerId
union all select 1 as weight, 3000497 as providerId
union all select 1 as weight, 3000496 as providerId
union all select 1 as weight, 3000499 as providerId
union all select 1 as weight, 3000498 as providerId
union all select 1 as weight, 3000461 as providerId
union all select 1 as weight, 3000460 as providerId
union all select 1 as weight, 3000463 as providerId
union all select 1 as weight, 3000462 as providerId
....




union all select -1000000 as weight, 3000984 as providerId),
markets as (
select distinct betting_type_id, event_part_id, event_id, provider_id, event_parent_id from `data-services-200001.x`
where date(event_original_start_time) BETWEEN @from AND @to and provider_id in (select providerId from weights)
and sport_id = @sportId
and is_live = @isLive
and event_parent_id = @tournamentId
)
select betting_type_id, event_part_id, event_id, eh.name, eh2.name as tournament_name, eh2.id as tournament_id, sum(w.weight) as total_weight, STRING_AGG(cast(w.providerId as STRING)) as providers
from markets m join weights w on m.provider_id = w.providerId
join `data-services-200001.y eh on eh.id = m.event_id
join `data-services-200001.y` eh2 on eh2.id = m.event_parent_id
where date(eh.original_start_time) BETWEEN @from and @to and eh.version = 0
and eh2.version = 0
group by betting_type_id, event_part_id, event_id, eh.name, eh2.name, eh2.id
having total_weight >= 3

I think the main problem comes from the WITH clause, but I am not pretty sure how can fix this in an elegant manner.

Any tips would be appreciated!

Upvotes: 2

Views: 1529

Answers (1)

Jose Gutierrez Paliza
Jose Gutierrez Paliza

Reputation: 1428

The error you're facing appears before the actual execution of the query and it's caused by its complexity. Regularly, the 3 main reasons that cause complex queries are:

  • a) Deeply nested and used repeatedly WITH clauses.
  • b) Deeply nested and used repeatedly VIEWs.
  • c) Lots of UNION ALL.

In your case, it happens because in the weights you have a lot of UNION ALL.

As I see your Query I would recommend you to Insert all the data into a table.

INSERT INTO `project.dataset.new_table`(weight, providerId)
VALUES(1,3000973)
VALUES(1,3000974)
VALUES(1,3000975)

After all the values are Inserted, you can use that table in the below query that you have. You need to change the values of project.dataset.new_table to your own.

markets as (
select distinct betting_type_id, event_part_id, event_id, provider_id, event_parent_id from `data-services-200001.x`
where date(event_original_start_time) BETWEEN @from AND @to and provider_id in (select providerId from `project.dataset.new_table`)
and sport_id = @sportId
and is_live = @isLive
and event_parent_id = @tournamentId
)
select betting_type_id, event_part_id, event_id, eh.name, eh2.name as tournament_name, eh2.id as tournament_id, sum(w.weight) as total_weight, STRING_AGG(cast(w.providerId as STRING)) as providers
from markets m join `project.dataset.new_table` w on m.provider_id = w.providerId
join `data-services-200001.y eh on eh.id = m.event_id
join `data-services-200001.y` eh2 on eh2.id = m.event_parent_id
where date(eh.original_start_time) BETWEEN @from and @to and eh.version = 0
and eh2.version = 0
group by betting_type_id, event_part_id, event_id, eh.name, eh2.name, eh2.id
having total_weight >= 3

Upvotes: 1

Related Questions