Error message while deploy a composer resource (GCP) with terraform

Question

I am having an error with a terraform code, while deploy a GCP composer resource:

• google_composer_environment.composer-beta: googleapi: Error 400: Property key must be of the form section-name. The section may not contain opening square brackets, closing square brackets or hyphens, and the name may not contain a semicolon or equals sign. The entire property key may not contain periods., badRequest

The issue arises while this GCP resource is being deployed: https://www.terraform.io/docs/providers/google/r/composer_environment.html

This is my code:

Variables.tf file:

variable "composer_airflow_version" {
  type = "map"
  default = {
    image_version="composer-1.6.1-airflow-1.10.1"
  }
}

variable "composer_python_version" {
  type = "map"
  default = {
    python_version="3"
  }
}

my-composer.tf file:

resource "google_composer_environment" "composer-beta" {
  provider= "google-beta"
  project = "my-proyect"
  name    = "${var.composer_name}"
  region  = "${var.region}"
  config {
    node_count = "${var.composer_node_count}"

  node_config {
    zone         = "${var.zone}"
    machine_type = "${var.composer_machine_type}"
    network      = "${google_compute_network.network.self_link}"
    subnetwork   = "${lookup(var.vpc_subnets_01[0], "subnet_name")}"
  }

  software_config {
    airflow_config_overrides="${var.composer_airflow_version}",
    airflow_config_overrides="${var.composer_python_version}",
  }
  }

  depends_on = [
    "google_service_account.comp-py3-dev-worker",
    "google_compute_subnetwork.subnetwork",
  ]
}

According to the error message, the root cause of the error seems be related to the software_config section in the terraform code. I understand that the variables "composer_airflow_version" and "composer_python_version" should be of type "map", therefore, I set up them as map format.

A really appreciate it, if someone could identify the cause of the error, and tell me the adjustment to apply. It is likely that I should apply a change in variables, but I don't know what it is. :-(

Thanks in advance, Jose

Answer 1

At google_composer_environment Terraform documentation I get:

airflow_config_overrides - (Optional) Apache Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation".

Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and cannot contain "=" or ";". Section and property names cannot contain characters: "." Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format. Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

From it I noticed that I was naming my overrides not following that suggested pattern. Like below:

airflow_config_overrides = {
        email_conn_id = "sendgrid_default"
        email_backend = "airflow.providers.sendgrid.utils.emailer.send_email"
        from_email    = "Darth <darth.vader@gmail.com>"
      }

when I changed it to this, it worked:

airflow_config_overrides = {
        "email-email_conn_id" = "sendgrid_default"
        "email-email_backend" = "airflow.providers.sendgrid.utils.emailer.send_email"
        "email-from_email"    = "Darth <darth.vader@gmail.com>"
      }

I find it weird that the first version worked multiple times. Then it stopped working.

Answer 2

Based on the documentations, airflow_config_overrides, pypi_packages, env_variables, image_version and python_version should be directly under software_config.

Variables.tf file:

variable "composer_airflow_version" {
  default = "composer-1.6.1-airflow-1.10.1"
}

variable "composer_python_version" {
  default = "3"
}

my-composer.tf file:

resource "google_composer_environment" "composer-beta" {
  provider= "google-beta"
  project = "my-proyect"
  name    = "${var.composer_name}"
  region  = "${var.region}"
  config {
    node_count = "${var.composer_node_count}"

    node_config {
      zone         = "${var.zone}"
      machine_type = "${var.composer_machine_type}"
      network      = "${google_compute_network.network.self_link}"
      subnetwork   = "${lookup(var.vpc_subnets_01[0], "subnet_name")}"
    }

    software_config {
      image_version  = "${var.composer_airflow_version}",
      python_version = "${var.composer_python_version}",
    }
  }

  depends_on = [
    "google_service_account.comp-py3-dev-worker",
    "google_compute_subnetwork.subnetwork",
  ]
}