CODEWITHSUNDEEP
streaminggoogle-bigquery
Max Lyakhovsky
Max Lyakhovsky

Reputation: 13

BigQuery - how to determine Data Availability

In regards to: https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability

What's the best way to programatically determine if a table's data is available after streaming?

I am getting unexpected results trying to fetch Rows and TotalRows with the following apis: Jobs.Query, Jobs.GetQueryResults, Tables.Get, Tabledata.List

Thanks.

Upvotes: 1

Views: 661

Answers (1)

Jordan Tigani
Jordan Tigani

Reputation: 26637

You can tell whether data is flushed on the table by executing the Tables.Get() API and looking at the streamingBuffer.oldestEntryTime value. This can be considered a high-water mark of data that has been flushed out of the buffer.

Any data before this timestamp should be available for copy, export, and list operations.

Also, I should clarify that data in the table is available for query immediately after streaming. It only is unavailable to table copy, export, and tabledata.list() operations. Yes, this is confusing, but yes, we're also working on addressing the problem.

For tables that haven't been streamed to before or recently, there is a warmup period where new streaming data won't show up.

See https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability for more information.

Upvotes: 3

Related Questions