How to create an EC2 machine with Packer?

Question

In AWS to make a new AMI, I usually run commands manually to verify that they're working, and then I image that box to create an AMI. But there are alternatives like packer.io, what's a minimal working example for using this service to make a simple customized AMI?

Answer 1

https://github.com/devopracy/devopracy-base/blob/master/packer/base.json There's a packer file that looks very similar to what I use at work for a base image. It's not tested, but we can go into it a bit. The base image is my own base - all services are built using it as a source ami. That way I control my dependencies and ensure there's a consistent os under my services. You could simply add cookbooks from the chef supermarket to see how provisioning a service works with this file, or use this as a base. As a base you would make a similar, less detailed build for the service and call this as the source ami.

This first part declares the variables I use to pack. The variables are injected before the build from a bash file which I DON'T CHECK IN TO SOURCE CONTROL. I keep the bash script in my home directory and source it before calling packer build. Note there's a cookbook path for the chef provisioner. I use the base_dir as the location on my dev box or the build server. I use a bootstrap key to build; packer will make it's own key to ssh if you don't specify one, but it's nice to make a key on aws and then launch your builds with it. That makes debugging packer easier on the fly.

"variables": {
"aws_access_key_id": "{{env `AWS_ACCESS_KEY`}}",
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
"ssh_private_key_file": "{{env `SSH_PRIVATE_KEY_FILE`}}",
"cookbook_path": "{{env `CLOUD_DIR`}}/ops/devopracy-base/cookbooks",
"base_dir": "{{env `CLOUD_DIR`}}"
},

The next part of the file has the builder. I use amazon-ebs at work and off work too, it's simpler to create one file, and often the larger instance types are only available as ebs. In this file, I resize the volume so we have a bit more room to install stuffs. Note the source ami isn't specified here, I lookup the newest version here or there. Ubuntu has a handy site if you're using it, just google ec2 ubuntu locator. You need to put in a source image to build on.

"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key_id`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-west-2",
"source_ami": "",
"instance_type": "t2.small",
"ssh_username": "fedora",
"ami_name": "fedora-base-{{isotime \"2006-01-02\"}}",
"ami_description": "fedora 21 devopracy base",
"security_group_ids": [ "" ],
"force_deregister": "true",
"ssh_keypair_name": "bootstrap",
"ssh_private_key_file": "{{user `ssh_private_key_file`}}",
"subnet_id": "",
"ami_users": [""],
"ami_block_device_mappings": [{
  "delete_on_termination": "true",
  "device_name": "/dev/sda1",
  "volume_size": 30
}],
"launch_block_device_mappings": [{
  "delete_on_termination": "true",
  "device_name": "/dev/sda1",
  "volume_size": 30
}], 
"tags": {
  "stage": "dev",
  "os": "fedora 21",
  "release": "latest",
  "role": "base",
  "version": "0.0.1",
  "lock": "none"
}
}],

It's very useful to tag your images when you start doing automations on the cloud. These tags are how you'll handle your deploys and such. fedora is the default user for fedora, ubuntu for ubuntu, ec2-user for amazon linux, etc. You can look those up in docs for your distro.

Likewise, you need to add a security group to this file, and a subnet to launch in. Packer will use the defaults in aws if you don't specify those but if you're building on a buildserver or a non-default vpc, you must specify. Force deregister will get rid of an ami with the same name on a successful build - I name by date, so I can iterate on the builds daily and not pile up a bunch of images.

Finally, I use the chef provisioner. I have the cookbook in another repo, and the path to it on the buildserver is a variable at the top. Here we're looking at chef-zero for provisioning, which is technically not supported but works fine with the chef client provisioner and a custom command. Beside the chef run, I do some scripts of my own, and follow it up by running serverspec tests to make sure everything is hunky dory.

  "provisioners": [


{
    "type": "shell",
    "inline": [

    ]
   },
   {
    "type": "shell",
    "script": "{{user `base_dir`}}/ops/devopracy-base/files/ext_disk.sh"
   },
   {
    "type": "shell",
    "inline": [
      "sudo reboot",
      "sleep 30",
      "sudo resize2fs /dev/xvda1"
   ]
   },
   {
    "type": "shell",
    "inline": [
      "sudo mkdir -p /etc/chef && sudo chmod 777 /etc/chef",
      "sudo mkdir -p /tmp/packer-chef-client && sudo chmod 777 /tmp/packer-chef-client"
    ]
   },
   {
    "type": "file",
    "source": "{{user `cookbook_path`}}",
    "destination": "/etc/chef/cookbooks"
   },
   {
    "type": "chef-client",
    "execute_command": "cd /etc/chef && sudo chef-client --local-mode -c /tmp/packer-chef-client/client.rb -j /tmp/packer-chef-client/first-boot.json",
    "server_url": "http://localhost:8889",
    "skip_clean_node": "true",
    "skip_clean_client": "true",
    "run_list": ["recipe[devopracy-base::default]"]
   },
   {
    "type": "file",
    "source": "{{user `base_dir`}}/ops/devopracy-base/test/spec",
    "destination": "/home/fedora/spec/"
   },
   {
    "type": "file",
    "source": "{{user `base_dir`}}/ops/devopracy-base/test/Rakefile",
    "destination": "/home/fedora/Rakefile"
   },
   {
    "type": "shell",
    "inline": ["/opt/chef/embedded/bin/rake spec"]
   },
   {
    "type": "shell",
    "inline": ["sudo chmod 600 /etc/chef"]
   }
  ]
}

Naturally there's some goofy business in here around the chmoding of the chef dir, and it's not obviously secure - I run my builds in a private subnet. I hope this helps you get off the ground with packer, which is actually an amazing piece of software, and good fun! Ping me in the comments with any questions, or hit me up on github. All the devopracy stuff is a WIP, but those files will probably mature when I get more time to work on it :P

Good Luck!