How to configure environment variables in Azure DevOps pipeline?

Question

I have an Azure Function (.NET Core) that is configured to read application settings from both a JSON file and environment variables:

var configurationBuilder = new ConfigurationBuilder()
                                .SetBasePath(_baseConfigurationPath)
                                .AddJsonFile("appsettings.json", optional: true)
                                .AddEnvironmentVariables()
                                .Build();

BuildAgentMonitorConfiguration configuration = configurationBuilder.Get<BuildAgentMonitorConfiguration>();

appsettings.json has the following structure:

{
  "ProjectBaseUrl": "https://my-project.visualstudio.com/",
  "ProjectName": "my-project",
  "AzureDevOpsPac": ".....",
  "SubscriptionId": "...",
  "AgentPool": {
    "PoolId": 38,
    "PoolName": "MyPool",
    "MinimumAgentCount": 2,
    "MaximumAgentCount": 10
  },
  "ContainerRegistry": {
    "Username": "mycontainer",
    "LoginServer": "mycontainer.azurecr.io",
    "Password": "..."
  },
  "ActiveDirectory": {
    "ClientId": "...",
    "TenantId": "...",
    "ClientSecret": "..."
  }
}

Some of these settings are configured as environment variables in the Azure Function. Everything works as expected:

The problem now is to configure some of these variables in a build pipeline, which are used in unit and integration tests. I've tried adding a variable group as follows and linking it to the pipeline:

But the environment variables are not being set and the tests are failing. What am I missing here?

Answer 1

Configure KeyVault Secrets in Variable Groups for FileTransform@1

The below will read KeyVault Secrets used in a Variable group and add them to the Environments Variables for FileTransform@1 to use

setup KeyVault Create Variable Group and import the values you want to use for the Pipeline.

In this example we used:

    - ConnectionStrings--Context
    - Cloud--AuthSecret
    - Compumail--ApiPassword

Setup names to match keyVault names: (you can pass these into the yml steps template)

#These parameters are here to support Library > Variable Groups > with "secrets" from a KeyVault #KeyVault keys cannot contain "_" or "." as FileTransform1@ wants #This script takes "--" keys and replaces them with "." and adds them into "env:" variables so Transform can do it's thing.

parameters:
- name: apiSecretKeys
  displayName: apiSecretKeys
  type: object
  default:
    - ConnectionStrings--Context
    - Cloud--AuthSecret
    - Compumail--ApiPassword
stages:
   - template: ./locationOfTemplate.yml
     parameters:
       apiSecretKeys: ${{ parameters.apiSecretKeys }}

... build api - publish to .zip file

Setup Variable groups on "job level"

variables:
  #next line here for pipeline validation purposes.. 
  - ${{if parameters.variableGroup}}:
      - group: ${{parameters.variableGroup}}
  #OR
  #- VariableGroupNameContinaingSecrets

Template file: (the magic)

parameters:
  - name: apiSecretKeys
    displayName: apiSecretKeys
    type: object
    default: []
steps:
  - ${{if parameters.apiSecretKeys}}:
  - powershell: |
      $envs = Get-childItem env:
      $envs | Format-List
    displayName: List Env Vars Before
  - ${{ each key in parameters.apiSecretKeys }}:
      - powershell: |              
          $writeKey = "${{key}}".Replace('--','.')
          Write-Host "oldKey  :" ${{key}}
          Write-Host "value   :" "$(${{key}})"
          Write-Host "writeKey:" $writeKey              
          Write-Host "##vso[task.setvariable variable=$writeKey]$(${{key}})"
        displayName: Writing Dashes To LowDash for ${{key}}      
   - ${{ each key in parameters.apiSecretKeys }}:
       - powershell: |
           $readKey = "${{key}}".Replace('--','.')
           Write-Host "oldKey  :" ${{key}}
           Write-Host "oldValue:" "$(${{key}})"
           Write-Host "readKey :" $readKey
           Write-Host "newValue:" ('$env:'+"$($readKey)" | Invoke-Expression)
         displayName: Read From New Env Var for ${{key}}      
   - powershell: |
       $envs = Get-childItem env:
       $envs | Format-List
     name: List Env Vars After adding secrets to env

7. Run Task - task: FileTransform@1

Enjoy ;)

Answer 2

If you are using bash then their example does not work, as they are referring incorrectly to the variables in the documentation. Instead it should be:

Store secret

#!/bin/bash
echo "##vso[task.setvariable variable=sauce]crushed tomatoes"
echo "##vso[task.setvariable variable=secret.Sauce;issecret=true]crushed tomatoes with garlic"

Retrieve secret

Wrong: Their example

#!/bin/bash
echo "No problem reading $1 or $SAUCE"
echo "But I cannot read $SECRET_SAUCE"
echo "But I can read $2 (but the log is redacted so I do not spoil the secret)"

Right:

#!/bin/bash
echo "No problem reading $(sauce)"
echo "But I cannot read $(secret.Sauce)"

Answer 3

I ran into this as well when generating EF SQL scripts from a build task. According to the docs, the variables you define in the "Variables" tab are also provided to the process as environment variables.

Notice that variables are also made available to scripts through environment variables. The syntax for using these environment variables depends on the scripting language. Name is upper-cased, . replaced with _, and automatically inserted into the process environment

For my instance, I just had to load a connection string, but deal with the case difference of the key between the json file and the environment:

var config = new ConfigurationBuilder()
              .AddJsonFile("appsettings.json", true, true)
              .AddEnvironmentVariables()
              .Build();

var connectionString = config["connectionString"] ?? config["CONNECTIONSTRING"];

Answer 4

I also have the same use case in which I want some environment variable to be set up using the azure build pipeline so that the test cases can access that environment variable to get the test passed. Directly setting the env variable using the EXPORT,ENV command does not work for the subsequent task so to have the environment variable set up for subsequent task follow the syntax as mentioned on https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch ie the task.set variable with the script tag

Correct way of setting ENV variable using build pipeline

- script: |
    echo '##vso[task.setvariable variable=LD_LIBRARY_PATH]$(Build.SourcesDirectory)/src/Projectname/bin/Release/netcoreapp2.0/x64'
  displayName: set environment variable for subsequent steps

Please be careful of the spaces as its is yaml. The above script tags set up the variable LD_LIBRARY_PATH (used in Linux to define path for .so files) to the directory defined.

This style of setting the environment variable works for subsequent task also , but if we set the env variable like mentioned below the enviroment variable will be set for the specefic shell instance and will not be applicable for subsequent tasks

Wrong way of setting env variable :

- script: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(Build.SourcesDirectory)/src/CorrectionLoop.HttpApi/bin/Release/netcoreapp2.0/x64
  displayName: Set environment variable

You can use the similar syntax for the setting up your environment variable.