CODEWITHSUNDEEP
pythongoogle-bigquery
rjh336
rjh336

Reputation: 128

How to write to the _default stream of a BigQuery table using the Write Storage API

I would like to use the BigQuery Write Storage API to stream single rows of data into a table. However, I do not want to create new write streams for each row.

According to the docs, each table has a default stream that can be written to. Google's explanation:

Additionally, every table has a special stream named ‘_default’ to which data can be written. This stream doesn’t need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.

I am wondering how to access this "_default" stream and write to it using the Python client?

I tried the following:

from google.cloud import bigquery_storage_v1

write_client = bigquery_storage_v1.BigQueryWriteClient()
stream_name = "projects/{projectId}/datasets/{datasetId}/tables/{tableName}/streams/_default"
stream = write_client.get_write_stream(name=stream_name)

But was met with the error: NotFound: 404 Requested entity was not found

EDIT 2022-02-24:

It turns out you cannot GET the _default stream using the client code in my example above. However, you can write to the _default stream using the name format projects/{projectId}/datasets/{datasetId}/tables/{tableName}/_default

Upvotes: 4

Views: 1877

Answers (2)

yiru
yiru

Reputation: 21

projects/{projectId}/datasets/{datasetId}/tables/{tableName}/streams/_default should work now.

It is recommended to use this sample to do your write: https://cloud.google.com/bigquery/docs/write-api-streaming

Upvotes: 0

rjh336
rjh336

Reputation: 128

It turns out you cannot GET the _default stream using the client code in my example above. However, you can write to the _default stream using the name format projects/{projectId}/datasets/{datasetId}/tables/{tableName}/_default

Upvotes: 2

Related Questions