CODEWITHSUNDEEP
linuxbashamazon-ec2amazon-web-services
clement
clement

Reputation: 4266

How to automatically snapshot a volume of an Amazon EC2 instance?

I'm trying a script to backup a volume automatically.

I follow this EBS-Snapshot.sh script as found on github: 

#!/bin/bash

# export EC2_HOME='/etc/ec2'  # Make sure you use the API tools, not the AMI tools
# export EC2_BIN=$EC2_HOME/bin
# export PATH=$PATH:$EC2_BIN
# I know all of the above is good to have solution, but not re-usable
# I have captured all of the above in a particular file and lemme execute it
source /etc/environment

PURGE_SNAPSHOT_IN_DAYS=10

EC2_BIN=$EC2_HOME/bin

# store the certificates and private key to your amazon account
MY_CERT='/path/to/certificate-file'
MY_KEY='/path/to/private-file'
# fetching the instance-id from the metadata repository
MY_INSTANCE_ID='your ec2-instance-id'

# temproary file
TMP_FILE='/tmp/rock-ebs-info.txt'

# get list of locally attached volumes via EC2 API:
$EC2_BIN/ec2-describe-volumes -C $MY_CERT -K $MY_KEY > $TMP_FILE
VOLUME_LIST=$(cat $TMP_FILE | grep ${MY_INSTANCE_ID} | awk '{ print $2 }')

sync

#create the snapshots
echo "Create EBS Volume Snapshot - Process started at $(date +%m-%d-%Y-%T)"
echo ""
echo $VOLUME_LIST
for volume in $(echo $VOLUME_LIST); do
   NAME=$(cat $TMP_FILE | grep Name | grep $volume | awk '{ print $5 }')
   DESC=$NAME-$(date +%m-%d-%Y)
   echo "Creating Snapshot for the volume: $volume with description: $DESC"
   echo "Snapshot info below:"
   $EC2_BIN/ec2-create-snapshot -C $MY_CERT -K $MY_KEY -d $DESC $volume
   echo ""
done

echo "Process ended at $(date +%m-%d-%Y-%T)"
echo ""

rm -f $TMP_FILE

#remove those snapshot which are $PURGE_SNAPSHOT_IN_DAYS old

I have the two files for X509 authentication, the instance ID but I don't understand the script and how to parameterise the volume that I want to backup.

I don't understand the first line (source) and the EC2_BIN. With that configuration, it lists all the volumes and makes a snapshot of all these...

For the comment of the snapshot, how can I change this line to add text?

DESC=$NAME-$(date +%m-%d-%Y)

I'm sorry to be a beginner but I don't understand the whole script

EDIT :

I get this error with this new code:

Creating Snapshot for the volume: ([ec2-describe-volumes]) with description: -03-13-2012 Snapshot info below: Client.InvalidParameterValue: Value (([ec2-describe-volumes])) for parameter volumeId is invalid. Expected: 'vol-...'. Process ended at 03-13-2012-08:11:35 –

And this is the code :

#!/bin/bash

#Java home for debian default install path:
export JAVA_HOME=/usr
#add ec2 tools to default path
#export PATH=~/.ec2/bin:$PATH


#export EC2_HOME='/etc/ec2'  # Make sure you use the API tools, not the AMI tools
export EC2_BIN=/usr/bin/
#export PATH=$PATH:$EC2_BIN
# I know all of the above is good to have solution, but not re-usable
# I have captured all of the above in a particular file and lemme execute it
source /etc/environment

PURGE_SNAPSHOT_IN_DAYS=60

#EC2_BIN=$EC2_HOME/bin

# store the certificates and private key to your amazon account
MY_CERT='cert-xx.pem'
MY_KEY='pk-xx.pem'
# fetching the instance-id from the metadata repository

MY_INSTANCE_ID=`curl http://169.254.169.254/1.0/meta-data/instance-id`

# temproary file
TMP_FILE='/tmp/rock-ebs-info.txt'

# get list of locally attached volumes via EC2 API:
$EC2_BIN/ec2-describe-volumes -C $MY_CERT -K $MY_KEY > $TMP_FILE

#VOLUME_LIST=$(cat $TMP_FILE | grep ${MY_INSTANCE_ID} | awk '{ print $2 }')
VOLUME_LIST=(`ec2-describe-volumes --filter attachment.instance-id=$MY_INSTANCE_ID | awk '{ print $2 }'`)

sync

#create the snapshots
echo "Create EBS Volume Snapshot - Process started at $(date +%m-%d-%Y-%T)"
echo ""
echo $VOLUME_LIST
echo "-------------"
for volume in $(echo $VOLUME_LIST); do
   NAME=$(cat $TMP_FILE | grep Name | grep $volume | awk '{ print $5 }')
   DESC=$NAME-$(date +%m-%d-%Y)
   echo "Creating Snapshot for the volume: $volume with description: $DESC"
   echo "Snapshot info below:"
   $EC2_BIN/ec2-create-snapshot -C $MY_CERT -K $MY_KEY -d $DESC $volume
   echo ""
done

echo "Process ended at $(date +%m-%d-%Y-%T)"
echo ""

rm -f $TMP_FILE

#remove those snapshot which are $PURGE_SNAPSHOT_IN_DAYS old

Upvotes: 5

Views: 15210

Answers (7)

Tarun Gupta
Tarun Gupta

Reputation: 6403

Create a rule that takes snapshots on a schedule. You can use a rate expression or a cron expression to specify the schedule. For more information

More information To create a rule

  1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

  2. In the navigation pane, choose Events, Create rule.

For Event Source, do the following:

a. Choose Schedule.

b. Choose Fixed rate of and specify the schedule interval (for example, 5 minutes). Alternatively, choose Cron expression and specify a cron expression (for example, every 15 minutes Monday through Friday, starting at the current time).

  1. For Targets, choose Add target and then select EC2 Create Snapshot API call.

  2. For Volume ID, type the volume ID of the targeted Amazon EBS volume.

  3. For AWS permissions, choose the option to create a new role. The new role grants the built-in target permissions to access resources on your behalf.

Choose Configure details.

For Rule definition, type a name and description for the rule.

Choose Create rule

Upvotes: 0

user855803
user855803

Reputation: 11

I think the best way now is to use AWS Lambda to take snapshots of your EC2 instances. you can find more details from this link

http://www.iwss.co.uk/ec2-instance-snapshot-through-aws-lambda-function-using-phyton-2-7/

Upvotes: 0

Sergio Troiano
Sergio Troiano

Reputation: 79

I came across with many people looking for a tool to administrate the EBS snapshots. I found several tools in internet but they were just scripts and incomplete solutions. Finally I decided to create a program more flexible, centralized and easy to administrate.

The idea is to have a centralized program to rule all the EBS snapshots (local to the instance or remotes)

I have created a small Perl program, https://github.com/sciclon/EBS_Snapshots

Some features: * Program runs in daemon mode or script mode (crontab)

  • You can chose only local attached volumes or remotes as well

  • You can define log file

  • You can define for each volume quantity of snapshots

  • You can define for each volume the frequency among them

  • Frequency and quantity will work like a "round-robin" when it reaches the limit removing the oldest snapshot.

  • you can readjust in one step the quantity I mean if you have 6 snapshots and you modify the quantity in 3 the process will readjust it automatically.

  • You can define a "prescript" execution, You can add your code to execute before executing the snapshot, for example you would like to try to umount the volume or stop some service, or maybe to check the instance load. The parent process will wait for the exit code, "0" means success, you can define if continue or not depending on the exit code.

  • You can define a "postscript" execution to execute any scrip after taking the snapshot (for example a email telling you about it)

  • You can add "Protected Snapshots" to skip the snapshot you define, I mean they will be in "read only" and they will never been erased.

  • you can reconfigure the script "on the fly" when it is running in daemon mode, the script accepts signals and IPC.

  • It has a "local-cache" to avoid requesting the API several times. You can add or modify any configuration in the config file and reload without killing the process.

Upvotes: 1

Havary Camara
Havary Camara

Reputation: 1

I don't know about you, but I prefer to make AMI instead snapshot. This script came from a idea from Craig, an employee of Amazon. They were developing a snapshot script called Arche. This script is simple - you mark a tag in an EC2 Instance and tag Ec2 are AMIed. I tested it in my environment. You can change the commands in this script to backup the snapshot, too.

Before you run this, config the linux environment variables with cert and pk keys.

#!/bin/bash
echo "AMI Backup is starting..."
echo "taking AMI Backup..."

day_of_year=$(date +%j)
week_of_year=$(date +%U)
week_of_year=$( printf "%.0f" $week_of_year )
year=$(date +%Y)

for INST in $(ec2-describe-instances --region=sa-east-1 --filter "tag:Backup=On" | awk '/^INSTANCE/ {print $2}')
do
        start_time=$(date +%R)
        ami=$(ec2-create-image $INST --name $INST$week_of_year --no-reboot | awk '{print $2}')
        ec2-create-tags $ami --tag Day_Year=$day_of_year > /dev/null
        ec2-create-tags $ami --tag Week_Year=$week_of_year > /dev/null
        ec2-create-tags $ami --tag Src_Instance=$INST > /dev/null
        ec2-create-tags $ami --tag Start_Time=$start_time > /dev/null
        end_time=$(date +%R)
        ec2-create-tags $ami --tag End_Time=$end_time > /dev/null
        echo "Created AMI $ami for volume $INST"
done

year=$(date +%Y)
expire_day=`expr $day_of_year  -  2`
expire_week=`expr $week_of_year  -  2`


echo "identifying AMI to be deleted"
for delete in $(ec2-describe-images --filter "tag:Week_Year=$expire_week" | awk '{ print $2;exit;}')
do
        ec2dereg $delete
        echo "deleted $delete"
done

Upvotes: 0

Sirch
Sirch

Reputation: 417

Heres a function i wrote in Ruby to snapshot all volumes on all instances in all regions. 

require 'aws-sdk'

def snapshot_all_attached_volumes(region)
  # For every instance in this region
  AWS::EC2.new(:region => region).instances.each do |instance|
    # get all the attached volumes
    instance.attachments.each do |mountpoint, attachment|
      # and create snapshots
      attachment.volume.create_snapshot(description = "Automated snapshot #{HOSTNAME}:#{$0}")
    end
  end
end

regions = AWS::EC2.regions.map(&:name)
regions.each do |region| 
  begin
    snapshot_all_attached_volumes(region)
    # delete_all_old_snapshots(region) 
  rescue
    puts "#{$!}"
  end 
end

Upvotes: 0

Sebastian Buckpesch
Sebastian Buckpesch

Reputation: 437

the above solution did not work completely for me. After I hour chat with the amazon support, I have now this working script, which will always create snapshots of all volumes attached to the current instance:

#!/bin/bash

# Set Environment Variables as cron doesn't load them
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export EC2_HOME=/usr
export EC2_BIN=/usr/bin/
export PATH=$PATH:$EC2_HOME/bin
export EC2_CERT=/home/ubuntu/.ec2/cert-SDFRTWFASDFQFEF.pem
export EC2_PRIVATE_KEY=/home/ubuntu/.ec2/pk-SDFRTWFASDFQFEF.pem
export EC2_URL=https://eu-west-1.ec2.amazonaws.com # Setup your availability zone here

# Get instance id of the current server instance
MY_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# get list of locally attached volumes 
VOLUMES=$(ec2-describe-volumes | grep ${MY_INSTANCE_ID} | awk '{ print $2 }')
echo "Instance-Id: $MY_INSTANCE_ID" 

    # Create a snapshot for all locally attached volumes
    LOG_FILE=/home/ubuntu/ebsbackup/ebsbackup.log
    echo "********** Starting backup for instance $MY_INSTANCE_ID" >> $LOG_FILE
    for VOLUME in $(echo $VOLUMES); do
        echo "Backup Volume:   $VOLUME" >> $LOG_FILE
        ec2-consistent-snapshot --aws-access-key-id ASDASDASDASD --aws-secret-access-key asdfdsfasdfasdfasdfasdf --mysql --mysql-host localhost --mysql-username root --mysql-password asdfasdfasdfasdfd --description "Backup ($MY_INSTANCE_ID) $(date +'%Y-%m-%d %H:%M:%S')" --region eu-west-1 $VOLUME
done
echo "********** Ran backup: $(date)" >> $LOG_FILE
echo "Completed"

I setup a cronjob in /etc/cron.d/ebsbackup

01 * * * * ubuntu /home/ubuntu/.ec2/myscriptname

This works pretty good for me... :-)

Hope this helps for you, Sebastian

Upvotes: 9

bwight
bwight

Reputation: 3310

Ok well,

  1. The first line where he runs (source). Thats the same as . /etc/environment. Anyways all he's doing is loading a file that has a list of environmental variables that amazon requires. At least this is what i assume.
  2. He's making this script much more complicated than it needs to be. He doesn't need to run the ec2-describe-instances command and save the output to a file then grep the output etc....
  3. You can put whatever you want for the DESC. You can just replace everything to the right of the = to whatever text you want. Just make sure to put quotes around it.

I would change two things about this script.

  1. Get the InstanceId at runtime in the script. Don't hard code it into the script. This line will work no matter where the script is running.

    MY_INSTANCE_ID=`curl http://169.254.169.254/1.0/meta-data/instance-id`

  2. Instead of calling ec2-describe-volumes and saving the output to a temp file etc... Just use a filter on the command and tell it which instance id you want.

    VOLUME_LIST=(`ec2-describe-volumes --filter attachment.instance-id=$MY_INSTANCE_ID | awk '{ print $2 }'`)

Upvotes: 4

Related Questions