How to use GitLab CICD variables in Terraform

Question

I'm trying to automate the Glue deployment using Gitlab and Terraform (running in a docker container in one of our Gitlab runner (EC2 Instance)). I know that I can have secret environment variables in GitLab, but I'm not sure how I can push those variables into my Terraform script - .gitlab-ci.yml file

I'm having set of terraform files places in the below format to automate my Glue deployment process.

Folder:

`->files/main.tf
   files/variables.tf
   files/provider.tf
.gitlab-ci.yml`    

In variables.tf file, i didnt use default option, instead of that I have placed variable in GitLab CICD variables.

In .gitlab-ci.yml file, i tried with below approaches to pass the CICD variables from GitLab but it is not helping me to figure it out.

1st approach using variables in .gitlab-ci.yml file as below,

`variables:
   TF_STATE_NAME: Glue_Terraform
   TF_VAR_aws_default_region: ${AWS_DEFAULT_REGION}
   TF_VAR_account_id: ${ACCOUNT_ID}
   TF_VAR_aws_access_key_id: ${AWS_ACCESS_KEY_ID}
   TF_VAR_aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
   TF_VAR_nc_prefix: ${NC_PREFIX}
   TF_VAR_aws_glue_role: ${AWS_GLUE_ROLE}
   TF_VAR_aws_s3_bucket: ${AWS_S3_BUCKET}
   TF_VAR_aws_s3_folder: ${AWS_S3_FOLDER}
   TF_VAR_aws_s3_tstate_bucket: ${AWS_S3_TSTATE}`    

2nd approach using export command in the script as below,

`script:
      - echo "cd $CI_PROJECT_DIR/terraform"
      - cd $CI_PROJECT_DIR/terraform
      - export TF_STATE_NAME="Glue_Terraform"
      - export TF_VAR_aws_default_region="REGION"
      - export TF_VAR_aws_access_key_id="KEY"
      - export TF_VAR_aws_secret_access_key="SECRET_ACCESS"
      - export TF_VAR_nc_prefix="PROJECT"
      - export TF_VAR_aws_glue_role="ROLE_ARN"
      - export TF_VAR_aws_s3_bucket="BUCKET_NAME"
      - export TF_VAR_aws_s3_folder="FOLDER_NAME"`    

I did a dry run to cross check whether the variable which I'm passing in script is valid or not without any environmental variable locally from Visio, it is working fine.

How can I pipe it to my Terraform scripts? Any ideas? I would need to read the secrets from GitLab's environment and pass it on to the Terraform scripts!

Answer 1

This has been fixed by configuring Environment and creating environment variable for DEV, QA and PROD as specific.

Thank you for who responded back to my query !!

Answer 2

You should have terraform environments if you follow this instruction

Example

Here is an example of a terraform plan in the GitLab CI pipeline.

.gitlab-ci.yml

image: hashicorp/terraform:latest

stages:
  - plan

variables:
  TF_VERSION: "1.8.5"
  TF_STATE_NAME: Glue_Terraform
  TF_VAR_aws_default_region: $AWS_DEFAULT_REGION
  TF_VAR_account_id: $ACCOUNT_ID
  TF_VAR_aws_access_key_id: $AWS_ACCESS_KEY_ID
  TF_VAR_aws_secret_access_key: $AWS_SECRET_ACCESS_KEY
  TF_VAR_nc_prefix: $NC_PREFIX
  TF_VAR_aws_glue_role: $AWS_GLUE_ROLE
  TF_VAR_aws_s3_bucket: $AWS_S3_BUCKET
  TF_VAR_aws_s3_folder: $AWS_S3_FOLDER
  TF_VAR_aws_s3_tstate_bucket: $AWS_S3_TSTATE

before_script:
  - terraform --version

apply:
  stage: plan
  script:
    - cd $CI_PROJECT_DIR/terraform
    - terraform init
    - terraform plan

Possible issues

• If selected protected variable, the variable is only available in pipelines that run on protected branches or protected tags. So if you need to run this pipeline on each branch, you should remove the protected variable from these variables.
• If you use environments in your gitlab ci. You should check if your variables have the same environment name as expected in your pipeline.

UPDATED 29-01-2024

Environment variables

If a variable only ever needs to be used in one specific environment, set it to only ever be available in that environment. For example, you can set a deploy token to only be available in the production environment. More information about variables can be read in this article.

Example

So first you should set up the environment for your variable, like in the attachment. So setup from the image for ACCESS_KEY_ID will be available only for protected branches and specific environment.

image: hashicorp/terraform:latest

stages:
  - plan

variables:
  TF_VERSION: "1.8.5"
  TF_STATE_NAME: Glue_Terraform
  TF_VAR_aws_default_region: $AWS_DEFAULT_REGION
  TF_VAR_account_id: $ACCOUNT_ID
  TF_VAR_aws_access_key_id: $AWS_ACCESS_KEY_ID
  TF_VAR_aws_secret_access_key: $AWS_SECRET_ACCESS_KEY
  TF_VAR_nc_prefix: $NC_PREFIX
  TF_VAR_aws_glue_role: $AWS_GLUE_ROLE
  TF_VAR_aws_s3_bucket: $AWS_S3_BUCKET
  TF_VAR_aws_s3_folder: $AWS_S3_FOLDER
  TF_VAR_aws_s3_tstate_bucket: $AWS_S3_TSTATE

before_script:
  - terraform --version

apply:
  stage: plan
  environment: production
  script:
    - cd $CI_PROJECT_DIR/terraform
    - terraform init
    - terraform plan