How to Share configuration between Terraform config and Packer templates?

Question

How do you share configuration between Terraform config and Packer templates?

At the moment I have a .tfvars file:

aws_region="us-east-1"
aws_access_key="non-sense"
...

and a Packer template:

{
  "variables": {
    "aws_access_key": "non-sense",
    "aws_secret_key": "key",
....
  },

Is there any possibility for both of them to use the same .env file or something similar?

Please don't narrow down it to only AWS configuration - let's say I need to reuse the name of the image.

Answer 1

If you are using Packer with HCL instead of JSON, you will find it has some similar approaches, e.g var files.

Packer (HCL) and TF allow us to define variables as below:

variable "template_name" {
  type = string
  default = "my-demo-packer"
}
variable "subnet_id" {
  type = string
}

Then use plain-text file to load the values: my-vars.pkrvars.hcl vs my-vars.tfvars:

template_name    = "my-template-001"
subnet_id        = "subnet-1234567"

if you want Packer/TF to auto-load this file, just add a step to rename it to my-vars.auto.pkrvars.hcl and my-vars.auto.tfvars inside your script.

Answer 2

There isn't a way to directly share normal configuration between these 2 systems but in your specific case that's okay because you shouldn't be hardcoding your credentials into configuration files anyway.

Both Packer and Terraform (along with nearly everything in the AWS ecosystem) will allow you to supply credentials through a chain of different ways with a precedence between them if multiple are set.

The AWS SDK for Go requires credentials (an access key and secret access key) to sign requests to AWS. You can specify your credentials in several different locations, depending on your particular use case. For information about obtaining credentials, see Setting Up.

When you initialize a new service client without providing any credential arguments, the SDK uses the default credential provider chain to find AWS credentials. The SDK uses the first provider in the chain that returns credentials without an error. The default provider chain looks for credentials in the following order:

1. Environment variables.

2. Shared credentials file.

3. If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.

4. If your application uses an ECS task definition or RunTask API operation, IAM role for tasks.

The SDK detects and uses the built-in providers automatically, without requiring manual configurations. For example, if you use IAM roles for Amazon EC2 instances, your applications automatically use the instance’s credentials. You don’t need to manually configure credentials in your application.

As a best practice, AWS recommends that you specify credentials in the following order:

1. Use IAM roles for tasks if your application uses an ECS task definition or RunTask API operation.

2. Use IAM roles for Amazon EC2 (if your application is running on an Amazon EC2 instance).

   IAM roles provide applications on the instance temporary security credentials to make AWS calls. IAM roles provide an easy way to distribute and manage credentials on multiple Amazon EC2 instances.

3. Use a shared credentials file.

   This credentials file is the same one used by other SDKs and the AWS CLI. If you’re already using a shared credentials file, you can also use it for this purpose.

4. Use environment variables.

   Setting environment variables is useful if you’re doing development work on a machine other than an Amazon EC2 instance.

This allows you to use your personal credentials when developing locally (either through setting environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or using the ~/.aws/credentials file) and then automatically use IAM roles when you run Packer or Terraform in AWS so STS can manage rotating short lived credentials for you.

---

If you wanted to share the ID of the image between Packer and Terraform then you should be using Terraform data sources so that Terraform can automatically select the latest image you have produced with Packer.

As an example you might have something like this:

packer.json

{
  "builders": [
    {
      "ami_name": "ubuntu/20.04/base/{{isotime | clean_resource_name}}",
      "source_ami_filter": {
        "filters": {
          "name": "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*",
          "root-device-type": "ebs",
          "virtualization-type": "hvm"
        },
        "most_recent": true,
        "owners": [
          "099720109477"
        ]
      }
    }
  ]
}

instance.tf

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/20.04/base/*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["self"]
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}

---

If you were hoping to avoid duplicating the string ubuntu/20.04/base/ between the above example Packer and Terraform code then unfortunately there isn't really an option other than to use variables for both Packer and Terraform and have some higher level system (such as a shell script) pass the same variable into each. This feels a bit awkward but would look something like this:

packer.json

{
  "variables": {
    "ami_base_name": ""
  },
  "builders": [
    {
      "ami_name": "{{user `ami_base_name`}}/{{isotime | clean_resource_name}}",
      "source_ami_filter": {
        "filters": {
          "name": "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*",
          "root-device-type": "ebs",
          "virtualization-type": "hvm"
        },
        "most_recent": true,
        "owners": [
          "099720109477"
        ]
      }
    }
  ]
}

instance.tf

variable "ami_base_name" {}

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["${var.ami_base_name}/*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["self"]
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}