Jenkins Terraform Secret from Credential Manager with TF_VAR_

Question

Everything that I've been reading says that I should be using the TF_VAR_ method of passing my client secret to Terraform, but I keep getting this error:

Error: No value for required variable

  on variables.tf line 1:
   1: variable "clientid" {

The root module input variable "clientid" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

Note that if I pass the values directly to apply with the -var flag, it works fine. So, this question is about how to use the TF_VAR_ method, not how to az login.

Jenkinsfile

//Jenkinsfile (Declarative Pipeline)
pipeline {
    agent { label 'docker-agent' }

    environment {
        AZURE_SUBSCRIPTION_ID='<......>'
        AZURE_TENANT_ID='<.......>'
        CONTAINER_REGISTRY='containerregistry'
        RESOURCE_GROUP='crrg'
        REPO="sftp01"
        IMAGE_NAME="sftptest"
        TAG="0.01"
    }
    
    stages {
        stage('build') {
            steps {
                //sh 'docker build -t containerregistry.azurecr.io/sftp01/sftptest:0.01 -f Dockerfile .'
                //sh 'echo built'
                
                withCredentials([usernamePassword(credentialsId: 'containerregistryCreds', passwordVariable: 'AZURE_CLIENT_SECRET', usernameVariable: 'AZURE_CLIENT_ID')]) {
                    
                    sh 'export TF_VAR_clientid=$AZURE_CLIENT_ID'
                    sh 'export TF_VAR_clientsecret=$AZURE_CLIENT_SECRET'
                    sh 'export TF_VAR_subscriptionid=$AZURE_SUBSCRIPTION_ID'
                    sh 'export TF_VAR_tenantid=$AZURE_TENANT_ID'
                    
                    //sh 'az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET -t $AZURE_TENANT_ID'
                    //sh 'az account set -s $AZURE_SUBSCRIPTION_ID'
                    //sh 'az acr login --name $CONTAINER_REGISTRY --resource-group $RESOURCE_GROUP'
                    //sh 'az acr build --image $REPO/$IMAGE_NAME:$TAG --registry $CONTAINER_REGISTRY --file Dockerfile . '
                    
                    sh 'terraform init'
                    sh 'terraform fmt'
                    sh 'terraform validate'
                    sh 'terraform apply -auto-approve -no-color'
                    
                    //If I pass the variables this way, it works fine.
                    //sh 'terraform apply -auto-approve -no-color -var clientid=$AZURE_CLIENT_ID -var clientsecret=$AZURE_CLIENT_SECRET -var subscriptionid=$AZURE_SUBSCRIPTION_ID -var tenantid=$AZURE_TENANT_ID'
                    
                    sh 'terraform show'
                    sh 'terraform state list'
                        }
            }
        }
    }
}

main.tf

# Configure the Azure provider
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0.2"
    }
  }
}

provider "azurerm" {
  client_id       = var.clientid
  client_secret   = var.clientsecret
  subscription_id = var.subscriptionid
  tenant_id       = var.tenantid
  features {}
}

resource "azurerm_resource_group" "resourceNameTFscopeOnly" {
  name     = "myTFResourceGroup"
  location = "westus2"
}

variables.tf

variable "clientid" {
  description = "Azure Client ID"
  type        = string
  sensitive   = true
}

variable "clientsecret" {
  description = "Azure Client Service Principal Secret"
  type        = string
  sensitive   = true
}

variable "subscriptionid" {
  description = "Azure Subscription ID"
  type        = string
  sensitive   = true
}

variable "tenantid" {
  description = "Azure Tenant ID"
  type        = string
  sensitive   = true
}

variable "testpassword" {
  description = "For testing (not relevant to question)"
  type        = string
  sensitive   = true
}

What am I doing wrong?

Answer 1

So as already stated in the other answer, you export the variables in one sh step but you run the Terraform commands in another sh step. Exported values do not survive across steps as these steps are standalone Bash processes. But you can run it all in one step:

sh """
export TF_VAR_...=...
terraform ...
"""

I think solution with env.TF_VAR_... = ... is still nicer though.

Answer 2

This is what I ended up doing. Thank you to Matt for answering the core question here. Also, in order to keep the tenantId and subscriptionId out of source control, I put those in their own usernamePassword credential in Jenkins:

//Jenkinsfile (Declarative Pipeline)
pipeline {
    agent { label 'docker-agent' }
    environment {
        //TF_LOG='DEBUG'
        TF_LOG_PATH='/home/jenkins/terraform-debug.log'
        CONTAINER_REGISTRY='ArcticaCR'
        RESOURCE_GROUP='crrg'
        REPO="sftp01"
        IMAGE_NAME="sftptest"
        TAG="0.01"
    }
    
    stages {
        stage('build') {
            steps {
                
                withCredentials([
                    usernamePassword(credentialsId: 'sftpServicePrincipalCreds', passwordVariable: 'TF_VAR_clientsecret', usernameVariable: 'TF_VAR_clientid'),
                    usernamePassword(credentialsId: 'AzureTenantSubscription', passwordVariable: 'TF_VAR_tenantid', usernameVariable: 'TF_VAR_subscriptionid'),
                    usernamePassword(credentialsId: 'passwordtestCreds', passwordVariable: 'TEST_PASSWORD', usernameVariable: 'TEST_USERNAME')
                ]) {
                     
                    sh 'az login --service-principal -u $TF_VAR_clientid -p $TF_VAR_clientsecret -t $TF_VAR_tenantid'
                    sh 'az account set -s $TF_VAR_subscriptionid'
                    sh 'az acr login --name $CONTAINER_REGISTRY --resource-group $RESOURCE_GROUP'
                    sh 'az acr build --image $REPO/$IMAGE_NAME:$TAG --registry $CONTAINER_REGISTRY --file Dockerfile . '
                    sh 'az logout'
                    
                    sh 'terraform init'
                    sh 'terraform fmt'
                    sh 'terraform validate'
                    
                    sh 'terraform apply -auto-approve -no-color -var testpassword=$TEST_PASSWORD'
     
                    sh 'terraform show'
                    sh 'terraform state list'

                        }
            }
        }
    }
}

Answer 3

The withCredentials block in Jenkins Pipeline will already export your variables to the environment within its scope. You can additionally modify the env object for additional environment variables, and access current environment variables (such as those you are assigning in the environment directive) from the env object. This is the reason Terraform errors on no values for the variables: none of them are being set in the environment correctly.

// assign password and username environment variables within block arguments
withCredentials([usernamePassword(credentialsId: 'containerregistryCreds', passwordVariable: 'TF_VAR_clientsecret', usernameVariable: 'TF_VAR_clientid')]) {
  script {
    // re-assign other environment variables from environment directive to env object within block scope           
    env.TF_VAR_subscriptionid = env.AZURE_SUBSCRIPTION_ID
    env.TF_VAR_tenantid = env.AZURE_TENANT_ID
    ...
  }
}