CODEWITHSUNDEEP
sqlgoogle-bigquerylarge-file-upload
Abdel Salam Majdi Abu Ghazaleh
Abdel Salam Majdi Abu Ghazaleh

Reputation: 1

Uploading large excel files to Big Query

how can i upload a large excel file and dataset (larger than 10 mb) to big query ? Can anyone help ?

I tried to research a way to do it, but everything i found was a little bit complicated. I just started working with SQL and I don't have much experience.

Upvotes: 0

Views: 119

Answers (1)

aameti
aameti

Reputation: 144

!!! WARNING prior to answer.

The below answer will incur costs. You can use --dry-run to estimate the number of bytes that are processed for bq and then combine it with Pricing calculator:

https://cloud.google.com/products/calculator/

What you are trying to achieve, is possible by loading a dataset from a google cloud storage bucket.

To make it easy for new users, there is a "Guide me" option from GCP on how to upload datasets larger than 10 mb.

GCP "Guide me" site

Don't forget to first estimate the costs.

An example of how this can be used using a Google Storage URI is referenced in this page:

  bq load \
    --source_format=CSV \
    mydataset.mytable \
    gs://mybucket/mydata.csv \
    ./myschema.json

Again, use pricing calculator to estimate costs.

To be able to create buckets on a specific project, you need specific IAM permissions, also referenced here

Upvotes: 0

Related Questions