Amazon ec2 python boto, spawning and connecting to server instance, cannot ssh into spawned instances

Question

I am producing python code to analyze the performance of various server AMI's offered by Amazon EC2. I am currently having issues trying to ssh into my spawned instances. I have successfully done so via their web interface, but cannot programmatically.

The code below spawns a single Red hat AMI with the programmatically produced security group and key pair (which is saved locally). After the instance is running I try to ssh into the instance using the saved key pair (after it has been chmod 400'd) but the ssh command freezes producing no output.

The code:

#!/usr/bin/env python

import sys
from boto.ec2 import EC2Connection

#Fill in with your respective keys
awsAccessKey = "" 
awsSecretKey = ""

#All AMI instance names from the free tier 
#In the EC2 panel, goto "instances" -> "launch instance" -> "free tier"
amiNameArr = ["ami-bba18dd2","ami-a25415cb","ami-e8084981","ami-ad184ac4","ami-7527031c"]
#Lets just use a varying set of AMI's 
amiDescArr = ["Amazon Linux","Red Hat Enterprise","SUSE Enterprise", 
              "Ubuntu Server 13.10","Microsoft Server 2012"]
#AMI Instance types, physical machine types that the AMIs run on; ti.micro only free one
#In order of optimizations: Micro, General, Memory, Storage, Compute 
amiInstTypesArr = ["t1.micro",
                   "m1.small","m1.medium","m1.large","m1.xlarge","m3.medium",
                   "m2.xlarge","m2.2xlarge","m2.4xlarge",
                   "hi1.4xlarge","hs1.8xlarge",
                   "c1.medium","c1.large","c3.large","c3.xlarge","c3.2xlarge"]

if __name__ == "__main__":
    from time import gmtime, strftime
    sessionStart = strftime("h%Hm%Ms%S", gmtime())
    #Connect to amazon AWS
    print("\nConnectiong to AWS, start time: " + sessionStart)
    awsConn = EC2Connection(awsAccessKey, awsSecretKey)
    connParms = awsConn.get_params()
    print("Connected with access key id: " + str(connParms['aws_access_key_id'])) 

    #Create a key pair for this session
    print("Creating key pair...")
    keyPairName = "AWSAnalysisKeyPair" + sessionStart
    awsKeyPair = awsConn.create_key_pair(keyPairName)
    awsKeyPair.save("~")
    print("Saved key pair: " + keyPairName)

    #Create a security group for all server instances to use
    print("Creating security group...")
    securityGroupName = "AWSAnalysisSecurityGroup" + sessionStart
    securityGroupDesc = "For access and analysis of programmatically spawned machines"
    awsSecGroup = awsConn.create_security_group(securityGroupName, securityGroupDesc)
    awsSecGroup.authorize('tcp',22,22,'0.0.0.0/0',awsSecGroup)
    awsSecGroup.authorize('tcp',80,80,'0.0.0.0/0',awsSecGroup)

    #Start spawning new server instances!
    #For each AMI, create all machine instance types we can
    print("Spawning instances...")
    for amiIndx in range(1, 2): #len(amiNameArr)):
        print("   AMI description: " + str(amiDescArr[amiIndx]))
        for typeIndx in range(0, 1): #len(amiInstTypesArr)):
            print("      starting machine: " + str(amiInstTypesArr[typeIndx]))
            awsConn.run_instances(
                amiNameArr[amiIndx],
                instance_type = amiInstTypesArr[typeIndx],
                security_groups = [securityGroupName],
                key_name = keyPairName,
                max_count = 1
            )

    #We now want to get information about each machine instance so we can analyze it


    #conn.get_all_instances() returns a list of Reservation objects
    from pprint import pprint
    print("All spawned instance information")
    reservations = awsConn.get_all_instances()
    instances = [i for r in reservations for i in r.instances]
    for i in instances:
        #pprint(i.__dict__) #Shows all possible instance info
        print("- id: " + str(i.__dict__['id']) + "\n"
              "   image: " + str(i.__dict__['image_id']) + "\n" + 
              "    type: " + str(i.__dict__['instance_type']) + "\n" + 
              "   state: " + str(i.__dict__['state']) + "\n" )

From viewing the online EC2 interface, I know that I am spawning an instance and it is running, also that it has the programmatically produced key pair and security group associated with it. Seeing as it has both of these associated with it, I would have to figure my issue lies in how I construct the key pair and security group.

Have I constructed the security group and key pair correctly? Are there any other reasons why I might not be able to ssh into these instances?

I also know that I am correctly trying to access the machine instance with ssh because I can successfully do so by spawning instances from the web interface and ssh'ing into them.

Answer 1

I just tested your script and - indeed - it is not working as expected :-)

First, it crashes on the last line. The 'state' information is now returned in an attribute named '_state'. So you you need to change line 76 as this :

  "   state: " + str(i.__dict__['_state']) + "\n" )

Secondly, your keypair, SG and instances are created, but if we look at the SG definition in the console, you will see

The "Source" is the name of the security group itself. This means that only other EC2 instances running in the same Security Group will be able to connect to these ports, not your laptop.

You should not add a SG object in your authorize API call. The modified code below will do it :

awsSecGroup.authorize('tcp',22,22,'0.0.0.0/0')
awsSecGroup.authorize('tcp',80,80,'0.0.0.0/0')

I just tested your script with the two modifications above and it works as expected.

$ ssh -i ~/AWSAnalysisKeyPairh09m55s41.pem ec2-user@184.72.84.162
Warning: Permanently added '184.72.84.162' (RSA) to the list of known hosts.
[ec2-user@ip-10-151-40-134 ~]$ uname -a
Linux ip-10-151-40-134 2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user@ip-10-151-40-134 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)

--Seb
AWS EMEA Technical Instructor