CODEWITHSUNDEEP
amazon-s3
Kakenx
Kakenx

Reputation: 1445

New to s3: How to setup automated backup in Amazon s3?

I'm pretty new to Amazon s3, so I just wanted to know how to setup an automated daily backup?

Thanks!

Upvotes: 0

Views: 2209

Answers (3)

N D
N D

Reputation: 737

You have three choices:

  1. Use glacier automated backups
  2. Use a third party service
  3. Back up to a separate s3 bucket using a script

If recovery speed is not an issue, you only need daily backups, and you are backing up large files then you should use the glacier life cycle rules to back up your bucket to glacier.

However when you do this the life cycle rules will warn you that this is not the most efficient solution if your file sizes are less than 256.0KB. You may get a warning like the following:

Transitioning Smaller Objects to Glacier May Increase Costs

The average size of objects in this bucket is 16.0KB* which is less than the recommended 256.0KB for lowering costs with Glacier Storage Class. Transitioning smaller objects to Glacier Storage Class can increase storage costs because an additional 32KB of Glacier index data and 8KB of S3 metadata are added.

If this is the case then you should backup your bucket to a "backup bucket." The following steps will help get around the glacier problem.

1.) Create a backup bucket here: https://console.aws.amazon.com/s3/home

2.) Create a user with a user policy like the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1426742219000",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetBucketPolicy",
                "s3:GetBucketAcl",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::test",
                "arn:aws:s3:::test/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetBucketPolicy",
                "s3:GetBucketAcl",
                "s3:GetObjectVersion",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::test-backup",
                "arn:aws:s3:::test-backup/*"
            ]
        }
    ]
}

This allows you to get the data from your app accessible buckets and to back up the data to your backup buckets but the backup user has no ability to delete data from the backup bucket which is important if your server is compromised and you should need to restore from these backups. You can create the policy here: https://console.aws.amazon.com/iam/home#policies and then add it to a user here https://console.aws.amazon.com/iam/home#users

3.) Enable versioning on your backup bucket here https://console.aws.amazon.com/s3/home

4.) You can use s3cmd to back up one bucket to another. Download and configure the tool from here: http://s3tools.org/s3cmd

#CONFIGURATION
export bucket_to_backup='test';
export backup_bucket='test-backup'; 
export AWS_ACCESS_KEY_ID=ASDFDSAFADSFDASF;
export AWS_SECRET_ACCESS_KEY=adsfdsaf86adsf5adsf568aadf75ads;
s3cmd --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --recursive sync s3://$bucket_to_backup s3://$backup_bucket;

You can set these commands up in a shell script and have them run in a crontab process.

5.) Once you have multiple backups you may want to restrict the number of versions that are stored in the backup version to save money on storage. You can do this by creating a life cycle rule on the bucket in the aws s3 console under bucket properties. There you can have it delete versions older than a certain amount of days.

6.) Once you have a variety of snap shots saving, you will need a way to restore your bucket with specific snap shots. Here is a PHP script that will help you do so:

export startSnapShotTimeStamp=1427577934;
export endSnapShotTimeStamp=1427577939;
#do the restore with php script
php awsUtil.php sync test test-backup $startSnapShotTimeStamp-$endSnapShotTimeStamp

The range of the timestamps restore your bucket to the most recent state between those two timestamps stored in your backup.

You can download the restoration script from here: http://boulderapps.co/post/backing-up-and-restoring-s3-to-a-versioned-bucket

Upvotes: 1

petertc
petertc

Reputation: 3911

I think duplicati is more suitable than s3cmd. s3cmd is command line managed tool for S3, but duplicati is designed for backup. see http://www.duplicati.com/

UPDATE: If you want simple backup without some advanced features in Linux, an alternative one is https://github.com/victorlin/avoid_disaster

Upvotes: 3

Rico
Rico

Reputation: 61521

The easiest way is to go is the s3cmd tool (S3 command line) in my opinion:

You can find information here: http://s3tools.org/s3cmd

You can just setup a cronjob that compresses all the files, including your database dump and them upload it to an S3 bucket using the s3cmd

[Edit]

On Debian to install it you can run:

sudo apt-get install s3cmd

Then you need a config file in your home directory named .s3cfg:

This is a sample:

[default]
access_key = <put AWS your access key here>
secret_key = <put AWS your secret access key here>
bucket_location = US

You can get an AWS access key using IAM and creating a user:

http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_SettingUpUser.html

Upvotes: 1

Related Questions