CODEWITHSUNDEEP
google-bigquery
Tauqeer Malik
Tauqeer Malik

Reputation: 11

Google Bigquery - Bulk Load

We have a csv file with 300 columns. Size is approx 250 MB. Trying to upload it to BQ through Web UI but the schema specification is hard work. I was anticipating BQ will identify file headers but it doesn't seems to be recognising unless I am missing something.Is there a way forward ?

Upvotes: 1

Views: 345

Answers (1)

romain-nio
romain-nio

Reputation: 1205

Yes, you have to write the schema by your own. Bigquery is not able to auto-infert it. If you have 300 columns, I sugget writing a script to automatically create the schema.

With the command-line tool (cf here) If you have some lines with the wrong/different schema, you can use the following option to continue for other records :

--max_bad_records : The maximum number of bad rows to skip before the load job

In your case if you want to skip the first line of headers, that can be the following :

bq load --skip_leading_rows=1 --max_bad_records=10000 <destination_table> <data_source_uri> [<table_schema>]

Upvotes: 3

Related Questions