CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerymultiprocessinggoogle-compute-engine
Tristan Bilot
Tristan Bilot

Reputation: 479

GCP compute engine bandwidth bottleneck with multiprocessing

I have recently created a tool to fetch BigQuery tables in parallel using multiprocessing backend. When this tool is running on a GCP compute engine, I do not see any bandwidth improvements when using multiple cores. However, referring to the GCE documentation, each vCPU can process an ingress of 2GB, for a maximum of 6 vCPUs. The machine I'm using is an e2-standard-32 :32 vCPU, 128 GB memory, and the used bandwidth peaks at ~100MiB/S, but I was expected to ~2GB/s * 6 vCPUs. When looking at the network bandwidth monitoring in GCP, I don't see a difference when fetching using multiprocessing.

Does anyone have an idea about how improve this bandwidth limitation imposed by compute engines?

Upvotes: 0

Views: 247

Answers (1)

MBHA Phoenix
MBHA Phoenix

Reputation: 2217

My guess is the limitation is originating from BigQuery service and not your VM bandwidth.

Upvotes: 3

Related Questions