Pass Azure Devops pipeline variable to template without it getting set to the literal value

Question

I have a pipeline with a template that plans terraform using TerraformTestV1@0

I'm currently setting a variable depending on the branch and want that to set the variable group it uses but it's setting the literal value and trying to find a variable group with that value. E.G It's searching for $(variable group) instead of transforming.

I've tried using $[variables.countryCode] and $(countryCode) but get the same outcome. I'm also using another variable in the same way but this one is getting transformed.

I know the countryCode variable is also getting set because I have a powershell task beforehand I tested that shows the output I was expecting.

Here's the pipeline I currently have setup

deploy.yml

name: $(BuildDefinitionName)_1.0$(Rev:.r)

trigger:
  tags:
    include:
    - 'refs/tags/uk-*'
    - 'refs/tags/us-*'
    - 'refs/tags/es-*'

pool: Default

variables:
- name: countryCode
  ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/es-') }}:
    value: es
  ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/us-') }}:
    value: us
  ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/uk-') }}:
    value: uk
- name: statefilename
  value: pipelinetest 

resources:
  repositories:
    - repository: templates
      type: git
      name: MyTemplateRepo

stages :
  - stage:
    jobs:
      - job: test
        steps:
              - task: PowerShell@2
                displayName: Split_Tag
                continueOnError: false
                inputs:
                  targetType: inline
                  script: | 
                    Write-host $(countryCode)

  - stage: Plan_Dev
    jobs:
    - template: terraform-plan2.yml@templates
      parameters:
        servicePrincipal: Development $[variables.countryCode]
        stateFileName: "$(statefilename).tfstate"
        variableGroup: Terraform $[variables.countryCode] Development Environment
        terraformVersion: '0.14.6'
        condition: ""

template:

parameters:
- name: 'servicePrincipal'
  default: 'CI Subscription'
  type: string
- name: 'stateFileName'
  type: string
- name: 'variableGroup'
  type: string
- name: 'condition'
  type: string
- name: 'terraformVersion'
  type: string
  default: "0.14.6"

jobs:
  - job: TerraformPlan
    condition: ${{ parameters.condition }}
    variables:
      - group: ${{ parameters.variableGroup }}
    steps:
          - task: TerraformInstaller@0
            displayName: Terraform Install
            inputs:
                terraformVersion: ${{ parameters.terraformVersion }}
          - task: replacetokens@3
            inputs:
                targetFiles: '**/*.tfvars'
                encoding: 'auto'
                writeBOM: true
                actionOnMissing: 'warn'
                keepToken: false
                tokenPrefix: '#{'
                tokenSuffix: '}'
                useLegacyPattern: false
                enableTransforms: false
                enableTelemetry: true
          - task: TerraformTaskV1@0
            displayName: Terraform init
            inputs:
                provider: 'azurerm'
                command: 'init'
                commandOptions: '-upgrade'
                backendServiceArm: ${{ parameters.servicePrincipal }}
                backendAzureRmResourceGroupName: $(backendResourceGroup)
                backendAzureRmStorageAccountName: $(backendStorageAccount)
                backendAzureRmContainerName: $(backendContainer)
                backendAzureRmKey: "$(short_location).$(tenant).$(short_environment).${{ parameters.stateFileName }}"
          - task: TerraformTaskV1@0
            displayName: Terraform Validate
            inputs:
                provider: 'azurerm'
                command: 'validate'
                backendAzureRmKey: ${{ parameters.servicePrincipal }}
          - task: TerraformTaskV1@0
            displayName: Terraform Plan
            inputs:
                provider: 'azurerm'
                command: 'plan'
                commandOptions: '-var-file="devops.tfvars"'
                environmentServiceNameAzureRM: ${{ parameters.servicePrincipal }}

This is the error I'm getting: The pipeline is not valid. Job TerraformPlan: Variable group Terraform $(countryCode) Development Environment could not be found. The variable group does not exist or has not been authorized for use.

Any ideas why this isn't getting transformed?

Answer 1

There are three syntaxes:

• Macro syntax: $(variableName)

  Macro-syntax is evaluated at runtime. So if you define the variable and then dynamically change it during the course of the pipeline, $(variableName) will always have the latest value.

• Runtime expressions: $[ expression ]

  Runtime expressions are tricky. They are evaluated once at runtime. An important note from the first few paragraphs in this article:

  Runtime expressions are intended as a way to compute the contents of variables and state

  In my interpretation and past experience, a runtime expression must be the entire right side of the equation. So a few examples:

  variables:
    myLiteral: 'hello'
    myVariable: $[ format('{0} world!', variables.myLiteral) ]
  
  steps:
  - powershell: write-host '$(myVariable)'
    condition: $[ contains( variables.myVariable, 'hello') ]
  

  Interesting observation, when used as a condition the $[ ] are not required:

  - powershell: write-host '$(myVariable)'
    condition: contains(variables.myVariable, 'hello')
  

• Compile-time expressions: ${{ expression }}

  Compile-time expressions are evaluated when the pipeline is compiling/expanding. Unlike runtime-expressions, there are fewer restrictions on where compile-time expressions can be used so they can appear in the middle of a literal. eg)

  parameters:
  - name: firstName
    type: string
  
  steps:
  - powershell: Write-Host 'Hello ${{ parameters.firstName }}'
  

  Very important: only very specific variables are available at compile-time! Refer to the Predefined Variables article as it has a table that lists which variables can be used at compile time.

  Also from the same article, emphasis is mine:

  The difference between runtime and compile time expression syntaxes is primarily what context is available. In a compile-time expression (${{ <expression> }}), you have access to parameters and statically defined variables. In a runtime expression ($[ <expression> ]), you have access to more variables but no parameters.

  The emphasis on statically defined variables refers to variables that are in the context at compile-time. It might seem like your countryCode is 'static' because it's declared upfront in the pipeline, but the reality is the actual value of the variable isn't resolved until after compilation has occurred.

---

From the example you've provided, the reason it doesn't work is because you're trying to pass a runtime value to a template parameter that results in a compile-time expansion of the variable group. At compile-time, the value of $(countryCode) is "$(countryCode)" which is why the variable group isn't found.

You'd need to make the parameter you're passing a compile-time evaluation:

  - stage: Plan_Dev
    jobs:
    - template: terraform-plan2.yml@templates
      parameters:
        servicePrincipal: Development $[variables.countryCode]
        stateFileName: "$(statefilename).tfstate"
        terraformVersion: '0.14.6'
        condition: ""
        ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/es-') }}:
           variableGroup: Terraform es Development Environment
        ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/us-') }}:
           variableGroup: Terraform us Development Environment
        ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/uk-') }}:
           variableGroup: Terraform uk Development Environment

In a similar topic, I think you might also discover the same problem with your servicePrincipal because Service Connections and variable groups are considered protected resources that are secured using the Approvals+Checks API which is evaluated before the pipeline runs. You might want to move the principal and the variable group inside the template and pass the countryCode as a compiled value.

---

Update:

After a little bit more experimentation, I was able to get ${{ variables['whatever'] }} and ${{ variables.whatever }} to resolve at compile time in the pipeline.yml.

Answer 2

You're using the wrong syntax to reference variables.

$[] is for runtime variables, which means variables that are set absolutely last in the process, after the YAML template is compiled. Unless you are specifically dealing with runtime variables, the safe assumption is that that syntax is incorrect.

The syntax you're looking for is ${{ variables.countryCode }}, or ${{ variables['countryCode'] }}. Any one of those should work. Compile-time variable references can be a bit tricky... some of those work in some circumstances, but not in others.

I've found the most consistently reliable syntax is ${{ variable.whatever }}.