CODEWITHSUNDEEP
terraformaws-glue
Jonito
Jonito

Reputation: 459

Terraform glue job doesn't create properly

i am using terraform and i don't get the right parameters to create my glue jobs. As i am not a terraform pro (i begin), i wonder how it works.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_job#glue_version

I have not the good updates on my glue job resource using those parameters:

resource "aws_glue_job" "job_name" {
  name     = "job_name"
  description  = "job-desc"
  role_arn = "${aws_iam_role.service-name.arn}"
  max_capacity  = 2
  max_retries  = 1
  timeout      = 60

  command {
    script_location = "s3://my_bucket"
    python_version  = "3"
  }

  default_arguments = {    
    "--job-language"    = "python"
    "--ENV"             = "env"
    "--spark-event-logs-path" = "s3://my_bucket"
    "--job-bookmark-option" = "job-bookmark-enable"
    "--glue_version" = "2.0"
    "--worker_type" = "G.1X"
    "--enable-spark-ui" = "true"
  }

  execution_property {
    max_concurrent_runs = 1
  }
}

Idon't know where and how put those params. Could you please help me ?

    "--glue_version" = "2.0"
    "--worker_type" = "G.1X"

Regards.

Upvotes: 1

Views: 5798

Answers (1)

Montgomery Watts
Montgomery Watts

Reputation: 4034

The glue_version and worker_type arguments go on the same level as the default_arguments block, not inside of it.

Once you move them out, your resource block may look like this:

resource "aws_glue_job" "job_name" {
  name         = "job_name"
  description  = "job-desc"
  role_arn     = "${aws_iam_role.service-name.arn}"
  max_capacity = 2
  max_retries  = 1
  timeout      = 60
  glue_version = "2.0"
  worker_type  = "G.1X"

  command {
    script_location = "s3://my_bucket"
    python_version  = "3"
  }

  default_arguments = {    
    "--job-language"          = "python"
    "--ENV"                   = "env"
    "--spark-event-logs-path" = "s3://my_bucket"
    "--job-bookmark-option"   = "job-bookmark-enable"
    "--enable-spark-ui"       = "true"
  }

  execution_property {
    max_concurrent_runs = 1
  }
}

EDIT

The version you are using, 2.30.0 doesn't support these arguments for the aws_glue_job resource.

The glue_version argument was not added until version 2.34.0 of the AWS Provider.

The worker_type argument was not added until version 2.39.0.

You will need to update the provider to support these arguments.

Upvotes: 5

Related Questions