CODEWITHSUNDEEP
bigdatagoogle-bigquerygoogle-cloud-storage
Badarinarayan
Badarinarayan

Reputation: 45

Uploading files automatically from Google Cloud to Big Query

How to automate the process of uploading .CSV files from Google Cloud onto Big Query.

Upvotes: 1

Views: 1375

Answers (1)

Pentium10
Pentium10

Reputation: 208022

Google Cloud Storage provides access and storage log files in CSV formats which can be directly imported into BigQuery for analysis. In order to access these logs, you must set up log delivery and enable logging. The schemas are available online, in JSON format, for both the storage access logs and storage bucket data. More information is available in the Cloud Storage access logs and storage data documentation.

In order to load storage and access logs into BigQuery from the command line, use a command such as:

bq load --schema=cloud_storage_usage_schema.json my_dataset.usage_2012_06_18_v0 gs://my_logs/bucket_usage_2012_06_18_14_v0

Upvotes: 1

Related Questions