CODEWITHSUNDEEP
google-cloud-platformstorage
user2736158
user2736158

Reputation: 419

upload greater than 5TB object to google cloud storage bucket

The max size limit for a single object upload is 5TB. How does one do a backup of a larger single workload? Say - I have a single file that is 10TB or more - that needs to be backed up to cloud storage?

Also, a related question - if the 10 TB is spread across multiple files (each file is less than 5TB) in a single folder, that shouldn't affect anything correct? A single object can't be greater than 5TB, but there isn't a limit on the actual bucket size. Say a folder containing 3 objects equal to 10TB, that upload will be automatically split across multiple buckets (console or gsutil upload)?

Thanks

Upvotes: 1

Views: 1280

Answers (2)

dstromberg
dstromberg

Reputation: 7167

Maybe look over http://stromberg.dnsalias.org/~dstromberg/chunkup.html ?

I wrote it to backup to SRB, which is a sort of precursor to Google and Amazon bucketing. But chunkup itself doesn't depend on SRB or any other form of storage.

Usage: /usr/local/etc/chunkup -c chunkhandlerprog -n nameprefix [-t tmpdir] [-b blocksize]
-c specifies the chunk handling program to handle each chunk, and is required
-n specifies the nameprefix to use on all files created, and is required
-t specifies the temporary directory to write files to, and is optional.  Defaults to $TMPDIR or /tmp
-b specifies the length of files to create, and is optional.  Defaults to 1 gigabyte
-d specifies the number of digits to use in filenames.  Defaults to 5

You can see example use of chunkup in http://stromberg.dnsalias.org/~strombrg/Backup.remote.html#SRB

HTH

Upvotes: 0

Joss Baron
Joss Baron

Reputation: 1524

You are right. The current limit on the size for individual objects is 5 TB. In this way you might split your file.

About the limitation on the Total Bucket size, there is no limit documented on this. Actually, in the overview says "Cloud Storage provides worldwide, highly durable object storage that scales to exabytes of data.". You might take a look into the best practices of GCS.

Upvotes: 1

Related Questions