CODEWITHSUNDEEP
pythongoogle-bigquerygoogle-cloud-storage
Marcelo Soares
Marcelo Soares

Reputation: 187

How can I load Cloud Storage data into Bigquery using Python?

I have some datasets (27 CSV files, separated by semicolons, summing 150+GB) that get uploaded every week to my Cloud Storage bucket.

Currently, I use the BigQuery console to organize that data manually, declaring the variables and changing the filenames 27 times. The first file replaces the entire previous database, then the other 26 get appended to it. The filenames are always the same.

How can I do it using Python?

Upvotes: 0

Views: 552

Answers (1)

Lena
Lena

Reputation: 51

Please, check out Cloud Functions functionality. It allows to use python. After the function is deployed, Cron Jobs can be created. Here is related question: Run a python script on schedule on Google App Engine

Also here is and article which describes, how to load data from Cloud Storage Loading CSV data from Cloud Storage

Upvotes: 2

Related Questions