CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-cloud-dataflowapache-beam
tezz
tezz

Reputation: 349

How to define max number of streams in Apache Beam using BigQuery Storage Read API

I want to read data from BigQuery and publish it to Pubsub. For this, I want to use Apache Beam and run this pipeline in Dataflow. And my BQ query takes a good amount of time to execute. So, want to reduce that time using BigQuery's Storage Read API which supports reading data using streams. Although Beam supports the usage of Storage API, I couldn't find documentation for setting the max number of streams. Can someone help me how to set number of streams in Apache Beam pipeline for reading data from BigQuery.

Refs: BQ Storage API Overview, Apache Beam BQ I/O

Upvotes: 0

Views: 156

Answers (1)

ningk
ningk

Reputation: 1383

All the tunable options for BigQueryIO can be found here. There doesn't seem to be an option for read.

You can double check this with [email protected]. If it's not supported, you may file a feature request at https://issues.apache.org/jira.

Upvotes: 0

Related Questions