CODEWITHSUNDEEP
google-bigquerygoogle-bi-engine
Anthony Liu
Anthony Liu

Reputation: 373

Mixing Big and Small Table processing in Bigquery BI Engine ? How long does BI Engine Cache the data?

Is it recommended to separate the projects that's used for ETL purpose ( Aggregating > 1TB data into a Table) and ( Querying data < 1TB ) Visualization Purpose.

Or will the BI Engine smartly does this? How long will BI Engine cache queried data if it does at all? e.g.

  1. None at all it will just use the reservation for in memory processing
  2. It will cache the data until it becomes full and discard the oldest data

Upvotes: 0

Views: 127

Answers (1)

Alexey Klishin
Alexey Klishin

Reputation: 184

It is recommended to separate projects the way you described. You then create BI Engine reservation for the visualization project to cache the data.

For your visualization project, BI Engine will store the most recent data in RAM and offload old data over time. There are no specific guarantees on time the data leaves in cache today - BI Engine will adapt to your workload.

Couple of things that may change in future:

  • Ability to choose which tables can use BI Engine reservation. That way you can store data in the same project with ETL, but pick tables that should be cached in BI Engine.
  • We may add some guarantee on how long the data is cached.

Upvotes: 0

Related Questions