CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-bi-engine
avloss
avloss

Reputation: 2646

How to ensure that BI Engine is running?

I've followed the tutorial found here https://cloud.google.com/bi-engine/docs/getting-started-data-studio I've also tried doing reservations in the region I'm using (London) for my own dataset. I did not see any difference "before" and "after", also I can't see any logs in the StackDriver. So I assume it's not working for whatever reason. But I have no idea how to start debugging it - it seems that it's supposed to somehow integrate automatically into BigQuery workflow, without any additional configurations, but that makes it very complicated to test and debug!

I'm not sure where to start. I assumed my "DataStuio Explorer" interactions would become N times faster once BI Engine is correctly enabled. Is that the right assumption?

Upvotes: 1

Views: 1187

Answers (1)

rsantiago
rsantiago

Reputation: 2099

BQ BI Engine is still in beta release, but there are some pieces of information in GCP that can answer your inquiries:

  • I did not see any difference "before" and "after", also I can't see any logs in the StackDriver. So I assume it's not working for whatever reason.

Once you create a reservation it will be available and costs will apply, not during the beta period but when it will be released to GA.

For more information on Stackdriver logs and metrics please refer to Monitoring BI Engine with Stackdriver.

For example, in the BigQuery Project logs you will find entries of the following type:

[TIMESTAMP] BigQuery BI Engine API CreateReservation europe

Those log entries indicate that the BI Engine is enabled and working; the capacity was reserved but not still used. The tutorial specifies certain circumstances under which the BI Engine is being used, see my next answer.

  • But I have no idea how to start debugging it - it seems that it's supposed to somehow integrate automatically into BigQuery workflow, without any additional configurations

The tutorial you are following indicates that "BI Engine uses the table and columns you configure to determine what data to cache. BI Engine only caches the columns you add to your report". Since the reservation is created in the US location, if you compare your results with other datasets in the same location the same capacity reservation will be used, and most probably no difference in time execution will be found. To verify this you may want to use a dataset in a location where no capacity reservation exist and compare execution times.

  • I assumed my "DataStuio Explorer" interactions would become N times faster once BI Engine is correctly enabled. Is that the right assumption?

That's right since BI Engine is an "in-memory analysis service" that will provide a faster response compared against the slots usage. Just consider that documentation also states that "If you run a query that produces query results larger than the size of your BI Engine capacity, the BI Engine self-tuning feature uses BigQuery slots to run the query", if your data always exceeds the capacity reservation, most probably you won't note any performance improvement.

Finally, notice that this is a beta product and has limited support.

Upvotes: 0

Related Questions