CODEWITHSUNDEEP
jsonamazon-web-servicesterraformterraform-provider-aws
cfelipe
cfelipe

Reputation: 350

How can I create an EMR cluster resource that uses spot instances without hardcoding the bid_price variable?

I'm using Terraform to create an AWS EMR cluster that uses spot instances as core instances.

I know I can use the bid_price variable within the core_instance_group block on a aws_emr_cluster resource, but I don't want to hardcode prices as I'd have to change them manually every time the instance type changes.

Using the AWS Web UI, I'm able to choose the "Use on-demand as max price" option. That's exactly what I'm trying to reproduce, but in Terraform.

on-demand

Right now I am trying to solve my problem using the aws_pricing_product data source. You can see what I have so far below:

data "aws_pricing_product" "m4_large_price" {
  service_code = "AmazonEC2"

  filters {
   field = "instanceType"
   value = "m4.large"
  }

  filters {
   field = "operatingSystem"
   value = "Linux"
  }

  filters {
   field = "tenancy"
   value = "Shared"
  }

  filters {
   field = "usagetype"
   value = "BoxUsage:m4.large"
  }

  filters {
   field = "preInstalledSw"
   value = "NA"
  }

  filters {
    field = "location"
    value = "US East (N. Virginia)"
  }
}

data.aws_pricing_product.m4_large_price.result returns a json containing the details of a single product (you can check the response of the example here). The actual on-demand price is buried somewhere inside this json, but I don't know how can I get it (image generated with http://jsonviewer.stack.hu/):

price-json

I know I might be able solve this by using an external data source and piping the output of an aws cli call to something like jq, e.g:

aws pricing get-products --filters "Type=TERM_MATCH,Field=sku,Value=8VCNEHQMSCQS4P39" --format-version aws_v1 --service-code AmazonEC2 | jq [........]

But I'd like to know if there is any way to accomplish what I'm trying to do with pure Terraform. Thanks in advance!

Upvotes: 1

Views: 383

Answers (2)

user12148208
user12148208

Reputation:

@cfelipe put that ${jsondecode(data.aws_pricing_product.m4_large_price.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD}" in a Locals

Upvotes: 1

ydaetskcoR
ydaetskcoR

Reputation: 56987

Unfortunately the aws_pricing_product data source docs don't expand on how it should be used effectively but the discussion in the pull request that added it adds some insight.

In Terraform 0.12 you should be able to use the jsondecode function to nicely get at what you want with the following given as an example in the linked pull request:

data "aws_pricing_product" "example" {
  service_code = "AmazonRedshift"

  filters = [
    {
      field = "instanceType"
      value = "ds1.xlarge"
    },
    {
      field = "location"
      value = "US East (N. Virginia)"
    },
  ]
}

# Potential Terraform 0.12 syntax - may change during implementation
# Also, not sure about the exact attribute reference architecture myself :)
output "example" {
  values = jsondecode(data.json_query.example.value).terms.OnDemand.*.priceDimensions.*.pricePerUnit.USD
}

If you are stuck on Terraform <0.12 you might struggle to do this natively in Terraform other than the external data source approach you've already suggested.

Upvotes: 1

Related Questions