CODEWITHSUNDEEP
google-cloud-platformgoogle-bigqueryarchitecturegoogle-cloud-storage
Anumeha
Anumeha

Reputation: 1

Fastest way to load data from Google Cloud Storage to Bigquery

We need to load data from Google cloud storage to Google Bigquery. What is the fastest and most cost efficient way?

  1. Dataflow -- Bigquery IO write
  2. GCStoBQOperator -- Cloud composer

Upvotes: 0

Views: 447

Answers (1)

Pentium10
Pentium10

Reputation: 208042

There is no fastest and cost efficient combination.

  1. Load from file, thats free, you can load files that could be 5TB of data. In non compressed mode is the fastest to load expect around 10-120 seconds for load time. Batch load docs.

  2. Load via Storage Write API, the default ingestion rate is 3GB per second. You can ingest up to 2TB per month for free, then there are some costs. I have a slide, docs are here.

  3. all other modes are legacy modes, and either are slow, or library implementations, that wrap some old APIs to do the import.

Upvotes: 0

Related Questions