CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-bigquerygoogle-kubernetes-engine
Fran Arenas
Fran Arenas

Reputation: 153

GKE Pod download speed of BigQuery API is very slow

I have an image that performs a query on BigQuery and then downloads the result (about 8GB) with the following code:

query_job = client.query(my_sql_query)
query_result = query_job.result()
data_table = query_result.to_arrow(progress_bar_type='tqdm')

When I run the code locally the download speed is OK (100 mbs, basically my computer actual bandwidth). The problem is that when I run It on GKE cluster the download is very slow (With tqdm I can see that it will be about 2 hours).

I checked the bandwidth of my pod with the python package speedtest-cli and it is near 500 mbs.

Upvotes: 2

Views: 714

Answers (1)

Fran Arenas
Fran Arenas

Reputation: 153

The problem was that I wasn't installed the package google-cloud-bigquery-storage inside the pod so I was using the google-cloud-bigquery package to perform the download.

Upvotes: 3

Related Questions