What should be the fastest method to load data in Google Cloud BigQuery?

Question

I want to load large size data, in google cloud bigQuery.

What are all the options at my hand (using UI and APIs) and what would be the fastest way?

TIA!

Answer 1

You can load data:

• From Google Cloud Storage
• From other Google services, such as DoubleClick and Google AdWords
• From a readable data source (such as your local machine)
• By inserting individual records using streaming inserts
• Using DML statements to perform bulk inserts
• Using a Google Cloud Dataflow pipeline to write data to BigQuery
• more formats at Introduction to Loading Data into BigQuery

Loading data into BigQuery from Google Drive is not currently supported, but you can query data in Google Drive using an external table.

You can load data into a new table or partition, you can append data to an existing table or partition, or you can overwrite a table or partition. For more information on working with partitions, see Managing Partitioned Tables.

When you load data into BigQuery, you can supply the table or partition schema, or for supported data formats, you can use schema auto-detection.

Each method is fast, if your data is large, you should go with the Google Cloud Storage.

When you load data from Google Cloud Storage into BigQuery, your data can be in any of the following formats:

• Comma-separated values (CSV)
• JSON (newline-delimited)
• Avro
• Parquet
• ORC (Beta)
• Google Cloud Datastore backups