CODEWITHSUNDEEP
apache-sparkapache-spark-sqlgoogle-bigquerygoogle-cloud-dataproc
foxwendy
foxwendy

Reputation: 2929

How does google Spark-BigQuery-Connector leverage BigQuery Storage API?

According to https://cloud.google.com/dataproc/docs/concepts/connectors/bigquery the connector uses BigQuery Storage API to read data using gRPC. However, I couldn't find any Storage API/gRPC usage in the source code here: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/tree/master/connector/src/main/scala

My questions are: 1. could anyone show me the source code where uses storage API & gprc call? 2. Does Dataset<Row> df = session.read().format("bigquery").load() work through GBQ storage API? if not, how to read from GBQ to Spark using BigQuery Storage API?

Upvotes: 1

Views: 495

Answers (1)

Igor Dvorzhak
Igor Dvorzhak

Reputation: 4457

  1. Spark BigQuery Connector uses only BigQuery Storage API for reads, you can see it here, for example.

  2. Yes, Dataset<Row> df = session.read().format("bigquery").load() works through BigQuery Storage API.

Upvotes: 4

Related Questions