Running multiple instances of the same application on a single machine in the cloud

Question

I have a worker application (non-web facing) which processes some jobs from an AWS SQS. I want to run multiple instances of this application but with different command-line arguments (maybe 30 processes in the end), so:

myApp arg1 arg2
myApp arg3 arg4

My infrastructure needs are pretty basic, but specifically I want each instance of the application to

1. Restart automatically if it goes down
2. Have some sort of basic monitoring + log retrieval
3. Support auto deployment from git
4. Run on a dedicated IP

What is the best infrastructure for hosting this? Preferred in Amazon ecosystem, but not required.

In Amazon:

• Amazon Kinesis + Amazon Lambda doesn't support the dedicated IP part
• Amazon EC2 + CodeDeploy and then I can configure UpStart or something on the EC2 image to keep the processes alive?
• Amazon EC2 Container Service (ECS) could run one docker container per process. I currently consider this my best option.

Not Amazon:

• Heroku supports "process types" which can run each process easily, but this requires one "dyno" per process. This seems to end up being ~$25/process which doesn't scale.

I'm just reaching out for ideas. Don't have much experience in Amazon. What other infrastructure options are particularly well suited to this problem?

Answer 1

Docker or rkt is your solution. You can be up and running in munites, all within one EC2 instance. Keep in mind that the number of your application instances will be limited by the number of IP addresses AWS allows for each instance type.

Take a look at how many IPv4 and IPv6 addresses you can have at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI. If you need more, ask AWS support to increase limits in advance.