CODEWITHSUNDEEP
google-bigqueryexportgoogle-cloud-storagegsutil
iamabhaykmr
iamabhaykmr

Reputation: 2011

Bigquery exporting tables in GCS of file size 8GB even using single wildcard URI to export the table in less than 1 gb chunksof file

I tried manually and with command line to export big query table having 140GB of data into files of size less than 1GB in GCS bucket. It created 168 files overall after export. All files from 1 to 167 are of less than 1GB but the last file is around 8GB for both case while exporting using command line or using big query interface.

Here is screenshot of GCS bucket.enter image description here

I followed Export bigquery table to GCS to export table into multiple file using single wildcard uri to split the exported table into chunks.

I want all exported files to be around 1 GB only. Can anybody help me with this? Thanks.

Upvotes: 0

Views: 1643

Answers (1)

Pentium10
Pentium10

Reputation: 208012

You read the documentation wrong.

There is no 1GB per file export configuration in BigQuery.

The 1GB what you have read is referring to the data size that you are trying to export.

If you are exporting more than 1 GB of data, you must export your data to multiple files. When you export your data to multiple files, the size of the files will vary.

So this tells that if your table is bigger than 1GB you must export to multiple files. But it DOESN'T tell you that the files will be smaller than 1GB, it tells the file size varies.

Upvotes: 1

Related Questions