CODEWITHSUNDEEP
amazon-web-servicesaws-cliamazon-emr
Hongbo Miao
Hongbo Miao

Reputation: 49954

How to specify spot instance type for the `aws emr create-cluster` command?

I created a EMR cluster by selecting "Use Spot purchasing option" for the Task nodes:

enter image description here

Then I copied the CLI command by clicking "View command for cloning cluster":

enter image description here

However, this command does not have any info for using the spot instance. If I create EMR cluster by this command, it will create Task nodes using on-demand instances:

aws emr create-cluster \
    --name hm-amazon-emr-cluster \
    --release-label emr-6.12.0 \
    --applications=Name Trino \
    --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","Name":"Primary","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}},{"InstanceCount":1,"InstanceGroupType":"CORE","Name":"Core","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}],"EbsOptimized":true}},{"InstanceCount":15,"InstanceGroupType":"TASK","Name":"Task","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}}]' \
    --log-uri s3n://hongbomiao-bucket/amazon-emr/logs/ \
    --service-role arn:aws:iam::8888888888888:role/service-role/AmazonEMR-ServiceRole-xxxxxxxxxxxxxxx \
    --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
    --auto-termination-policy '{"IdleTimeout":7200}' \
    --step-concurrency-level 10 \
    --region us-west-2

Below is prettified version of --instance-groups:

[
  {
    "InstanceCount": 1,
    "InstanceGroupType": "MASTER",
    "Name": "Primary",
    "InstanceType": "r5.xlarge",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ]
    }
  },
  {
    "InstanceCount": 1,
    "InstanceGroupType": "CORE",
    "Name": "Core",
    "InstanceType": "r5.xlarge",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ],
      "EbsOptimized": true
    }
  },
  {
    "InstanceCount": 15,
    "InstanceGroupType": "TASK",
    "Name": "Task",
    "InstanceType": "r5.xlarge",
    "EbsConfiguration": {
      "EbsBlockDeviceConfigs": [
        {
          "VolumeSpecification": {
            "VolumeType": "gp2",
            "SizeInGB": 32
          },
          "VolumesPerInstance": 2
        }
      ]
    }
  }
]

I tried to create two clusters with "Use Spot purchasing option" for the Task nodes selected and unselected then copy out the command by clicking "View command for cloning cluster" button. And I got same contents.

How to specify spot instance type for the aws emr create-cluster command? Thanks!

Upvotes: 0

Views: 308

Answers (1)

Lionia Vasilev
Lionia Vasilev

Reputation: 12757

Add the BidPrice property to the TASK instance group configuration. That is the maximum bidding price for a Spot Instance: it implicitly shows that you would like to use Spot Instances. Please see the documentation for command create-cluster for more information.

[BidPrice] - If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD.

Additionally, it seems like there is a bug in the export to CLI function. You can send a bug report in the AWS Management Console: click on the question mark icon on the page's header -> Send Feedback -> Set the type to "Report an issue".

Upvotes: 2

Related Questions