CODEWITHSUNDEEP
logginggoogle-bigquerygoogle-cloud-storage
Jamie
Jamie

Reputation: 1

Loading logs from Google Cloud Storage bucket into BigQuery

My goal is to get all my (GCS created) logs from Google Cloud Storage into BigQuery, using the BigQuery web interface. Since logs are created hourly I like to import them all at once to analyze.

So for example gs://logging-bucket/images.mysite.net_storage_2013_03_31_07_00_00_0001a_v0 , gs://logging-bucket/images.mysite.net_storage_2013_03_31_07_00_00_0002a_v0 and so on.

I get this error when getting just one file:

Errors:

 Line:1 / Column:1, Expected '{' found '"'
 Line:2 / Column:1, Expected '{' found '"' 
 Line:3 / Column:1, Expected '{' found '"'
 Line:4 / Column:1, Expected '{' found '"'
 Line:5 / Column:1, Expected '{' found '"'
 Line:6 / Column:1, Expected '{' found '"'
 Line:7 / Column:1, Expected '{' found '"'

What I did is create a new dataset, selected data from gs://logging-bucket/images.mysite.net_storage_2013_03_31_07_00_00_0001a_v0 then added this schema http://storage.googleapis.com/pub/cloud_storage_usage_schema_v0.json and submitted this.

Similar questions on here and Google ing this didn't help me a lot.

Please, thank you.

Upvotes: 0

Views: 881

Answers (1)

Jordan Tigani
Jordan Tigani

Reputation: 26617

Can you give an example of a row? Is the data in json format or csv? From the error, it sounds like it is trying to import JSON data, but a json row shouldn't start with '"' -- it should start with '{'. Is this CSV data? If so, you should set the sourceFormat to CSV (or leave it empty) instead of NEWLINE_DELIMITED_JSON.

Upvotes: 0

Related Questions