CODEWITHSUNDEEP
amazon-ec2mapreduceelastic-map-reduceemr
user1804287
user1804287

Reputation:

EMR - Leverage using spot instances

I know that we can bid on spot instances and get them at lower prices than that of regular instances, but with spot instances there is the risk of your instances being taken back.

I want to know that is there any way we can ensure that they are taken away only when our jobs are finished?

Upvotes: 2

Views: 734

Answers (1)

Amar
Amar

Reputation: 12020

The only way to 'ensure' it would be to bid a very very very very high value!

But realistically, as you know, if you request spot instances, keep in mind that if the current spot price exceeds your max bid, either instances will not be provisioned or will be removed from the current job flow. To hedge the complete lose of a jobflow, multiple instance groups can be created where the CORE group is a smaller complement of traditional on-demand systems and the TASK group is the group of spot instances. In this configuration, the TASK group will only benefit the mapper phases of a job flow as work from the TASK group is "hand back up" to the CORE group for reduction.

So say if you have to run a job which would ideally need 40 slave machines, then you can have say 10 machines(CORE group) as the traditional instance while other 30 as spot instances(TASK group). The syntax for creating the multiple instance groups is below:

elastic-mapreduce --create --alive --plain-output
...
--instance-group master --instance-type m1.small --instance-count 1 \
--instance-group core   --instance-type m1.small --instance-count 10 \
--instance-group task   --instance-type m1.small --instance-count 30 --bid-price 0.018

Upvotes: 5

Related Questions