CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-glacier
Omroth
Omroth

Reputation: 1129

Zip up folders as I upload them to AWS Glacier

I currently use aws sync to sync my large data repository into Glacier

aws s3 sync . s3://backup --storage-class DEEP_ARCHIVE

My repository is very large and has millions of files, so I pay a large charge when uploading.

Ideally I would like to zip every directory at the point of uploading, compare/sync with the Glacier archive, then remove the zip - because I don't have the local space to keep the zips around.

What approach would you use for this?

Thanks

Upvotes: 0

Views: 2302

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 270059

You would be responsible for performing the zip and compare operations.

The ability to zip is not available as part of the AWS CLI or Amazon S3 Glacier.

Fewer zip files would make it easier to backup, but a large zip file is more costly (in time and money) to restore. If you only wish to restore a small portion of the zip file, you would still need to retrieve the whole zip file. Therefore, you will need to find the "happy medium" between easy backup and easy restoration.

The easiest solution is probably to use traditional backup software that knows how to work with Amazon S3. For example, Cloud Backup Solution for Small Business | CloudBerry Lab. Such vendors are well-versed in the best way to backup and restore software. It would be cheaper in terms of time to use such a product.

Upvotes: 4

Related Questions