CODEWITHSUNDEEP
terraform
Pierre-Alexandre
Pierre-Alexandre

Reputation: 765

Terraform: loop over directory to create a single resource

I am trying to create a single GCP Workflows using Terraform (Terraform Workflows documentation here). To create a workflow, I have defined the desired steps and order of execution using the Workflows syntax in YAML (can also be JSON).

I have around 20 different jobs and each of theses jobs are on different .yaml files under the same folder, workflows/. I just want to loop over the /workflows folder and have a single .yaml file to be able to create my resource. What would be the best way to achieve this using Terraform? I read about for_each but it was primarily used to loop over something to create multiple resources rather than a single resource.

workflows/job-1.yaml

- getCurrentTime:
    call: http.get
    args:
        url: https://us-central1-workflowsample.cloudfunctions.net/datetime
    result: currentDateTime

workflows/job-2.yaml

- readWikipedia:
    call: http.get
    args:
        url: https://en.wikipedia.org/w/api.php
        query:
            action: opensearch
            search: ${currentDateTime.body.dayOfTheWeek}
    result: wikiResult

main.tf

resource "google_workflows_workflow" "example" {
  name          = "workflow"
  region        = "us-central1"
  description   = "Magic"
  service_account = google_service_account.test_account.id
  source_contents = YAML FILE HERE

Upvotes: 2

Views: 4766

Answers (2)

Martin Atkins
Martin Atkins

Reputation: 74259

Terraform has a function fileset which allows a configuration to react to files available on disk alongside its definition. You can use this as a starting point for constructing a suitable expression for for_each:

locals {
  workflow_files = fileset("${path.module}/workflows", "*.yaml")
}

It looks like you'd also need to specify a separate name for each workflow, due to the design of the remote system, and so perhaps you'd decide to set the name to be the same as the filename but with the .yaml suffix removed, like this:

locals {
  workflows = tomap({
    for fn in local.workflow_files :
    substr(fn, 0, length(fn)-5) => "${path.module}/workflows/${fn}"
  })
}

This uses a for expression to project the set of filenames into a map from workflow name (trimmed filename) to the path to the specific file. The result then would look something like this:

{
  job-1 = "./module/workflows/job-1.yaml"
  job-2 = "./module/workflows/job-2.yaml"
}

This now meets the requirements for for_each, so you can refer to it directly as the for_each expression:

resource "google_workflows_workflow" "example" {
  for_each = local.workflows

  name            = each.key
  region          = "us-central1"
  description     = "Magic"
  service_account = google_service_account.test_account.id
  source_contents = file(each.value)
}

Your question didn't include any definition for how to populate the description argument, so I've left it set to hard-coded "Magic" as in your example. In order to populate that with something reasonable you'd need to have an additional data source for that, since what I wrote above is already making full use of the information we get just from scanning the content of the directory.

Upvotes: 5

Ayush Verma
Ayush Verma

Reputation: 68

resource "google_workflows_workflow" "example" {
    
    # count for total iterations
    count = 20 

    name          = "workflow"
    region        = "us-central1"
    description   = "Magic"
    service_account = google_service_account.test_account.id
    
    # refer to file using index, index starts from 0
    source_contents = file("${path.module}/workflows/job-${each.index}.yaml")

}

Upvotes: 0

Related Questions