BigQuery batch load execution time and frequency

Question

I'm considering switching from Streaming insert to Batch loading for my BigQuery insertion. However, I can't find some information I'd need.

• Time: Let's say I have a csv file with 1 million record. Roughly, how much time does a load take? Are we talking hours or minutes?
• Frequency: Can I have a load job every 10 minutes or so?
• Concurrency: Can two jobs happen at the same time, or do I need to wait for the previous one to finish?

Answer 1

For uncompressed CSV files with 1 million records expect around 10-15 seconds of processing time. But the question to put here is where the file is stored, and how long it will be taken to be uploaded, as that can be more than the above time section.

We have successfully imported in 2 minutes CSV files up to 5TB of data in it.

Your project can run up to 100,000 load jobs per day. Failed load jobs count toward this limit.

But be aware: Your project can make up to 5,000 partition modifications per day to an ingestion-time partitioned table.

Your project can make up to 30,000 partition modifications per day for a column-partitioned table.

Your project can run up to 50 modifications per partitioned table every 10 seconds.

You can have multiple load jobs in parallel.

BigQuery is really petabyte scaling, so large jobs fit nice.

Although it has some Quotas, but they are quite decent, see them here.