CODEWITHSUNDEEP
google-bigquery
Avi Zloof
Avi Zloof

Reputation: 2973

Big query is to slow

I am just starting with biquery, my DB is small (10K of rows 1 table) and my queries are simple count and group by. Its takes and average of 3-4 sec per request but sometimes its jumps to 10 and event 15sec I am querying from amazon linux server in Irland using the BQ tool.

Is it possible to get results faster (under 1sec) so I will be able to present my webpages faster.

Upvotes: 7

Views: 13970

Answers (3)

dylanvanw
dylanvanw

Reputation: 3341

BigQuery released a new feature to address this problem.

It is specifically designed for short queries to run much faster. It has the potential to make it possible to use bigquery data in user facing dashboards or applications.

To run a query using a short optimized query, you simply have the select it as the query mode when executing a query.

enter image description here

Upvotes: 0

Pentium10
Pentium10

Reputation: 208042

1) Big Query is a highly scalable database, before being a "super fast" database. It's designed to process HUGE amount of data distributing the processing among several different machines using a technique named Dremel. Because it's designed to use several machines and parallel processing, you should expect to have super-scalability with a good performance.

2) BigQuery is an asset when you want to analyze billions of rows.

For example: analyzing all the wikipedia revisions in 5-10 seconds isn't bad, is it? But even a much smaller table would take about the same time, even if has 10k rows.

3) Under this size, you'll be better off using more traditional data storage solutions such as Cloud SQL or the App Engine Datastore. If you want to keep SQL capability, Cloud SQL is the best guess.

Sybase IQ is often installed in a single database and it doesn't use Dremel. That said, it's going to be faster than Big Query in many scenarios...as designed.

4) Certainly the performance differ from a dedicated environment. You get your dedicated environment for 20K$ a month.

Upvotes: 6

Javier Ramirez
Javier Ramirez

Reputation: 4032

That's the expected behaviour. In BigQuery you are using a shared infrastructure, so depending on the use at the moment you will get better or worse response time. Actually batch queries (those not needing interactivity) are encouraged and rewarded by not adding up to your quota.

You typically don't use BigQuery as your main database to show data in your web application. Depending on what you want to do, BigQuery can be a Big Data storage and you should have another intermediate store where you could store computed results to display to your users. Or maybe in your use case you don't really need BigQuery and there is a better solution.

In any case, you are not going to be able to avoid a few seconds wait (even if you go Premium, you get more guarantees about the service, but in no case a service fast enough as to be your main backend for a webapp)

Upvotes: 6

Related Questions