Updating terraform (.tf) files using Python

Question

I am trying to update terraform templates using python, although I am having some issues trying to find a module (like PyYAML to update yml files).

My aim is to update the terraform files present in a git repo, and create a pull request, which when merged would trigger a pipeline to implement the changes.

Example .tf file:

variable "variable_1" {}
variable "variable_2" {}

locals {
  temp_locals = 0
}

resource "aws_iam_role" "MY_AWS_ACCOUNT" {
  name = "MY_AWS_ACCOUNT"

  assume_role_policy = <:user/user.name1"
        ]
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "aws_iam_role_policy_attachment" {
}

Want to add arn of user.name2 in this file.

Martin Atkins · Accepted Answer

There is currently no Python implementation of Terraform's underlying syntax HCL which can support surgical modification to an existing file, because that requires preserving information that a typical HCL parser will not retain, such as comments and the ordering of arguments within blocks.

Another option is to use a lesser-used feature of Terraform called Override Files, which allow you to write a new file that overrides part of an existing file. This mechanism exists specifically for this sort of situation where you want to use the content from a file written by humans but override part of it with content generated by a machine.

If you take your given example code as example.tf then you can put another file on disk next to it called example_override.tf.json with the following content:

{
  "resource": {
    "aws_iam_role": {
      "MY_AWS_ACCOUNT": {
        "assume_role_policy": "{
	"Version": "2012-10-17",
	"Statement": [
	  {
		"Action": "sts:AssumeRole",
		"Condition": {
		  "Bool": {
			"aws:MultiFactorAuthPresent": "true"
		  }
		},
		"Principal": {
		  "AWS": [
			"arn:aws:iam:::user/user.name2"
		  ]
		},
		"Effect": "Allow",
		"Sid": ""
	  }
	]
  }"
      }
    }
  }
}

Because this additional file uses Terraform's alternative JSON syntax, it is easy to generate from any language that has a JSON serializer available, including Python. Terraform itself will read both files and take the assume_role_policy from the override file as overriding the corresponding one in the base file, as described in Merging Behavior. You can then leave the original HCL file unmodified, avoiding issues with interfering with human-oriented formatting decisions in that file.

The Go implementation of HCL, which is the canonical one and the one that Terraform itself uses, has a package hclwrite which is a specialized API for directly modifying existing HCL source code while preserving all of the unmodified tokens and their ordering. Therefore a program written in Go could potentially achieve a result like what you were looking for using that package:

package main

import (
    "fmt"
    "log"

    hcl "github.com/hashicorp/hcl/v2"
    "github.com/hashicorp/hcl/v2/hclwrite"
    "github.com/zclconf/go-cty/cty"
)

func main() {
    // In a real program, use ioutil.ReadFile to read a file from disk.
    // But for this example, I will use a const string.
    src := []byte(oldSource)
    f, diags := hclwrite.ParseConfig(src, "example.tf", hcl.Pos{Line: 1, Column: 1})
    if diags.HasErrors() {
        log.Fatal(diags)
    }

    for _, block := range f.Body().Blocks() {
        if block.Type() != "resource" {
            continue
        }
        if labels := block.Labels(); len(labels) < 2 || labels[0] != "aws_iam_role" || labels[1] != "MY_AWS_ACCOUNT" {
            continue
        }

        block.Body().SetAttributeValue("assume_role_policy", cty.StringVal(newPolicy))
    }

    // In a real program, maybe write these bytes to a file on disk.
    fmt.Println(string(f.Bytes()))
}

const oldSource = `
variable "variable_1" {}
variable "variable_2" {}

locals {
  temp_locals = 0
}

resource "aws_iam_role" "MY_AWS_ACCOUNT" {
  name = "MY_AWS_ACCOUNT"

  assume_role_policy = <:user/user.name1"
        ]
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "aws_iam_role_policy_attachment" {
}
`

const newPolicy = `
{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Condition": {
          "Bool": {
            "aws:MultiFactorAuthPresent": "true"
          }
        },
        "Principal": {
          "AWS": [
            "arn:aws:iam:::user/user.name2"
          ]
        },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  }
`

The output of this program does, however, demonstrate one of the limitations of automatic generation of a language that's primarily designed to be written and read by humans:

variable "variable_1" {}
variable "variable_2" {}

locals {
  temp_locals = 0
}

resource "aws_iam_role" "MY_AWS_ACCOUNT" {
  name = "MY_AWS_ACCOUNT"

  assume_role_policy = "
{
	"Version": "2012-10-17",
	"Statement": [
	  {
		"Action": "sts:AssumeRole",
		"Condition": {
		  "Bool": {
			"aws:MultiFactorAuthPresent": "true"
		  }
		},
		"Principal": {
		  "AWS": [
			"arn:aws:iam:::user/user.name2"
		  ]
		},
		"Effect": "Allow",
		"Sid": ""
	  }
	]
  }
"
}

resource "aws_iam_role_policy_attachment" "aws_iam_role_policy_attachment" {
}

As humans, we can make subjective decisions about how to format things for readability, but hclwrite has just defaulted to using the quoted string form for this string, which in this case is a poor choice that a human author would likely not have made.

Updating terraform (.tf) files using Python

Answers (1)

Related Questions