CODEWITHSUNDEEP
google-bigquerygoogle-cloud-storage
searain
searain

Reputation: 3301

Google bigquery export big table to multiple objects in Google Cloud storage

I have two bigquery tables, bigger than 1 GB.

To export to storage,

https://googlecloudplatform.github.io/google-cloud-php/#/docs/google-cloud/v0.39.2/bigquery/table?method=export

$destinationObject = $storage->bucket('myBucket')->object('tableOutput_*');
$job = $table->export($destinationObject);

I used wild card.

Strange things is one bigquery table is exported to 60 files each of them with 3 - 4 MB size.

Another table is exported to 3 files, each of them close to 1 GB, 900 MB.

The codes are the same. The only difference is in the case that the table exported to 3 files. I put them into a subfolder.

The one exported to 60 files are one level above the subfolder.

My question is how bigquery decided that a file will be broken into dozens smaller files or just be broken into a few big files (as long as each file is less than 1GB)?

Thanks!

Upvotes: 1

Views: 548

Answers (1)

Victor Mota
Victor Mota

Reputation: 1269

BigQuery makes no guarantees on the sizes of the exported files, and there is currently no way to adjust this.

Upvotes: 2

Related Questions