CODEWITHSUNDEEP
google-cloud-platformterraformterraform-provider-gcp
PL Sergent
PL Sergent

Reputation: 468

Re-run the "program" of an external data source on every plan (refresh state) terraform

I'm using an external data source in order to perform a cURL command using the program argument :

data "external" "curl_zip" {
    program = ["bash", "-c", "curl", ...]
}

I'm running the Terraform in a pipeline so I need to retrieve the data on every terraform plan.

It seemed to work well until I created a new resource requiring the curl to be performed. But it looks like Terraform is only refreshing and so doesn't do the program command after the first plan:

data.external.curl_zip["something.json"]: Refreshing state... [id=-]

My question is : is there a way to re-run the program argument on every plan even during refresh ?

PS : I already tried to use a null_resource instead with a local-exec, turned out to not be the solution here because (for some reason) I also need to use a archive_file data source to create zips files so my GCP app engines resource can read them, and the local-exec is being executed after the terraform apply, which doesn't work since the data source is being refreshed or created during the plan.

Upvotes: 3

Views: 2913

Answers (2)

PL Sergent
PL Sergent

Reputation: 468

Update

Thank you very much for your response and sorry for not getting back to you earlier. So for some reason I had a hard time using the data source, and it seems that after an apply the data source was saved into the state : terraform state list. So the following plan did not recreate the data source that I was using to execute the curl.

So I got back to the null_resource solution. And it's a little bit complicated, but previously I had a problem with the curl request that have made the zip unusable for my GAE resource. So I had to use an archive_file on top of the curl. But this couldn't work because a data source is being treated during the plan and the local-exec of the null_resource is being executed during the apply.

Anyway so I fixed my curl so I don't need the archive_file data source. I also needed to change the interpreter to ["/bin/bash", "-c"] to make this work. Moreover, I used a trigger to always run the curl during each apply.

Here is my resource :

resource "null_resource" "curl_zip" {
    for_each = local.json_data
    provisioner "local-exec" {
        command = "curl -H 'API_KEY' -sLo ./path/to/zip"
        interpreter = ["/bin/bash", "-c"]
    }

    triggers = {
        always_run = timestamp()
    }
}

Upvotes: 1

jdsalaro
jdsalaro

Reputation: 388

It seems to me like you're over-complicating things. What, exactly, are you trying to achieve? With all things Terraform it's usually better to ask how can I achieve so and so? than how can I get my Terraform code to work?.

Is the following what you're after?

data "external" "hello" {
    program = ["bash", "-c", "echo 'Hello World!' > helloworld.txt; echo -n '{\"hello\":\"world!\"}'"]
}

resource "null_resource" "world" {
  provisioner "local-exec" {
    command = "echo '${data.external.hello.result["hello"]}'"
  }
}

As you can see from the timestamps in the following output, helloworld.txt is being generated five times, once every time Terraform plan is invoked:

jdsalaro$ for i in {1..5} ;do terraform plan; ls -lah --full-time helloworld.txt ;done \
| grep helloworld.txt | cut -d ' ' -f 7,9

00:04:18.610304219 helloworld.txt
00:04:19.902246088 helloworld.txt
00:04:21.226186506 helloworld.txt
00:04:22.574125835 helloworld.txt
00:04:23.886066774 helloworld.txt

I uploaded the whole example here just in case.

Upvotes: 3

Related Questions