Azure devops appsettings.json variable substitution from Key Vault in pipelines not working

Question

I have a .Net 6 web application that I am building in devops. I am trying to substitute settings in the appsettings.json, but somehow can't get it working.

What I have:

1. Azure Key vault set up, with a secret with the name: Configuration--ConnectionStrings--ConnectionString
2. Pipelines have access to this Key Vault with a Variable Group
3. I have tried the transform task in both build and release pipeline
4. I have tried the option in the IIS Web app deploy task for substitution

1 - All is fine

2 - All is fine

3 - I have tried this task in build and release with the following configuration:

steps:
- task: FileTransform@1
  displayName: 'File Transform: '
  inputs:
    folderPath: '$(System.DefaultWorkingDirectory)/**/WebAppFront.zip'
    fileType: json
    targetFiles: '**/appsettings.json'

The logs look OK for transform:

2022-03-17T10:04:32.9753812Z ##[section]Starting: File Transform: 
2022-03-17T10:04:33.0157518Z ==============================================================================
2022-03-17T10:04:33.0158091Z Task         : File transform
2022-03-17T10:04:33.0158579Z Description  : Replace tokens with variable values in XML or JSON configuration files
2022-03-17T10:04:33.0159048Z Version      : 1.198.0
2022-03-17T10:04:33.0159390Z Author       : Microsoft Corporation
2022-03-17T10:04:33.0159938Z Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform
2022-03-17T10:04:33.0160522Z ==============================================================================
2022-03-17T10:04:33.6732027Z [command]C:\azagent\A1\_work\_tasks\FileTransform_8ce97e91-56cc-4743-bfab-9a9315be5f27\1.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_2021667764440822 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:33.7472675Z 
2022-03-17T10:04:33.7679746Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:33.7687417Z 
2022-03-17T10:04:33.7689291Z Scanning the drive for archives:
2022-03-17T10:04:33.7693972Z 1 file, 21535247 bytes (21 MiB)
2022-03-17T10:04:33.7839085Z 
2022-03-17T10:04:33.7855780Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6203882Z --
2022-03-17T10:04:36.6205010Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:36.6206577Z Type = zip
2022-03-17T10:04:36.6238261Z Physical Size = 21535247
2022-03-17T10:04:36.6250740Z 
2022-03-17T10:04:36.6282342Z Everything is Ok
2022-03-17T10:04:36.6282941Z 
2022-03-17T10:04:36.6283536Z Folders: 24
2022-03-17T10:04:36.6284339Z Files: 112
2022-03-17T10:04:36.6284868Z Size:       58919697
2022-03-17T10:04:36.6288304Z Compressed: 21535247
2022-03-17T10:04:36.6338841Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:36.7353081Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_2021667764440822\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:36.7444592Z JSON variable substitution applied successfully.
2022-03-17T10:04:40.1757797Z ##[section]Finishing: File Transform: 

And also for the rest of the log like getting keyvault (done before transform):

2022-03-17T10:04:32.0237340Z ##[section]Starting: Download secrets: my-key-vault
2022-03-17T10:04:32.0691326Z ==============================================================================
2022-03-17T10:04:32.0691706Z Task         : Azure Key Vault
2022-03-17T10:04:32.0691934Z Description  : Download Azure Key Vault secrets
2022-03-17T10:04:32.0692142Z Version      : 2.200.0
2022-03-17T10:04:32.0692355Z Author       : Microsoft Corporation
2022-03-17T10:04:32.0692657Z Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
2022-03-17T10:04:32.0693010Z ==============================================================================
2022-03-17T10:04:32.7133186Z SubscriptionId: my-subscription-id.
2022-03-17T10:04:32.7145990Z Key vault name: my-key-vault.
2022-03-17T10:04:32.7152879Z Downloading secret value for: Configuration--ConnectionStrings--ConnectionString.
2022-03-17T10:04:32.9707096Z ##[section]Finishing: Download secrets: my-key-vault

The appsettings.json looks like this:

{
    "Configuration": {
        "ApplicationName": "Lorem ipsum",
        "Logging": {
            "LogLevel": {
                "Default": "Information",
                "Microsoft": "Warning",
                "Microsoft.Hosting.Lifetime": "Information"
            }
        },
        "ConnectionStrings": {
            "ConnectionString": ""
        }
    }
}

But the connectionstring stays empty. As I understood from reading a lot of blogs and documentation this should be working. The double hyphen -- in the key name, should translate to a dot, so Json path expression is correct (I think): Configuration.ConnectionStrings.ConnectionString

4. The IIS Web app deploy task, with transform gives similar result. So an empty connectionstring. Even though it explicitly states it updated the json in the zippackage.

Logs for that task:

2022-03-17T10:04:43.3419710Z ##[section]Starting: IIS Web App Deploy
2022-03-17T10:04:43.3940659Z ==============================================================================
2022-03-17T10:04:43.3941018Z Task         : IIS web app deploy
2022-03-17T10:04:43.3941276Z Description  : Deploy a website or web application using Web Deploy
2022-03-17T10:04:43.3941526Z Version      : 0.198.0
2022-03-17T10:04:43.3941721Z Author       : Microsoft Corporation
2022-03-17T10:04:43.3942065Z Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/iis-web-app-deployment-on-machine-group
2022-03-17T10:04:43.3943670Z ==============================================================================
2022-03-17T10:04:44.0803824Z [command]C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\7zip\7zip\7z.exe x -oC:\azagent\A1\_work\_temp\temp_web_package_39200019901712446 C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:44.1320582Z 
2022-03-17T10:04:44.1365755Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2022-03-17T10:04:44.1393221Z 
2022-03-17T10:04:44.1427445Z Scanning the drive for archives:
2022-03-17T10:04:44.1436856Z 1 file, 23620753 bytes (23 MiB)
2022-03-17T10:04:44.1451518Z 
2022-03-17T10:04:44.1587844Z Extracting archive: C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1556142Z --
2022-03-17T10:04:47.1556846Z Path = C:\azagent\A1\_work\r1\a\_Vind\drop\WebAppFront.zip
2022-03-17T10:04:47.1558328Z Type = zip
2022-03-17T10:04:47.1558787Z Physical Size = 23620753
2022-03-17T10:04:47.1559027Z 
2022-03-17T10:04:47.1566021Z Everything is Ok
2022-03-17T10:04:47.1566315Z 
2022-03-17T10:04:47.1566617Z Folders: 35
2022-03-17T10:04:47.1567095Z Files: 112
2022-03-17T10:04:47.1567474Z Size:       58919916
2022-03-17T10:04:47.1567818Z Compressed: 23620753
2022-03-17T10:04:47.2311338Z Applying JSON variable substitution for **/appsettings.json
2022-03-17T10:04:47.3655684Z Applying JSON variable substitution for C:\azagent\A1\_work\_temp\temp_web_package_39200019901712446\Content\D_C\a\1\s\Vind\WebAppFront\obj\Release\net6.0\PubTmp\Out\appsettings.json
2022-03-17T10:04:47.3751003Z JSON variable substitution applied successfully.
2022-03-17T10:04:50.9446139Z [command]"C:\azagent\A1\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.198.0\node_modules\azure-pipelines-tasks-webdeployment-common\MSDeploy3.6\MSDeploy3.6\msdeploy.exe" -verb:sync -source:package='C:\azagent\A1\_work\r1\a\temp_web_package_8577780759906015.zip' -dest:auto -setParam:name='IIS Web Application Name',value='www.mydomain.com' -enableRule:DoNotDeleteRule
2022-03-17T10:04:51.7358688Z Info: Updating file (www.mydoain.com\appsettings.json).
2022-03-17T10:04:51.7839580Z Total changes: 1 (0 added, 0 deleted, 1 updated, 0 parameters changed, 2281 bytes copied)
2022-03-17T10:04:51.8207636Z ##[section]Finishing: IIS Web App Deploy

What am I doing wrong? The only thing I noticed when I was echoing out the variable it got cut off on a ; in the connectionstring, but even when I tried some simple value it was not working.

Answer 1

I managed to resolve this issue automatically without the need to manage all the variables by yourself (and keep the KeyVault).

See this answser :

https://stackoverflow.com/questions/61469253/azure-devops-pipelines-releases-access-azure-key-vault-secret-nested/73635255#73635255

Answer 2

I ended up solving it by creating variables in the pipelines and using the Key Vault secrets as the value.

Keyvault secret name: 'MyConnectionSecret'

You have to use the dot notation to specify a path. See my question for the json structure. Name of var in Dev ops:

Configuration.ConnectionStrings.ConnectionString

Value of var in Dev ops:

$(MyConnectionSecret)

And of course in the Library you have to add the variable group based on the Keyvault