How to run Azure CLI tasks from an Azure DevOps Pipeline on a Self-Hosted Windows Agent?

Question

Situation

My self-hosted Windows Agent runs a pipeline from Azure DevOps. To manage resources in Azure I want to use an Azure CLI task. The AzureCLI task fails, even though Azure CLI is installed in a prior step.

I have two scripts that run from my pipeline.

• (1) Install Azure CLI --> Success
• (2) Run Azure CLI commands --> Fails without running ANY of the code inside, even "Hello, World!" does not get executed.

2021-03-05T14:50:02.5986237Z ##[error]Azure CLI 2.x is not installed on this machine.
2021-03-05T14:50:02.6391547Z ##[error]Script failed with error: Error: Unable to locate executable file: 'az'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

Microsoft says

• (1) After you install new software on an agent, you must restart the agent for the new capability to show up in the pool so that the build can run.
• (2) After the installation is complete, you will need to reopen PowerShell to use the Azure CLI.

The AzureCLI task is unable to find the installed Azure CLI executable. How can I fix this so that I can run the AzureCLI task?

What I tried already

• Setting the PATH of Azure CLI via PowerShell. Path is set but the Powershell task for Azure CLI task fails.
• Running AzureCLI commands directly in my installation script, which works but I need to log in to Azure with separate credentials while I want to use the service principal defined in my AzureCLI task.
• Restarting Microsoft Agent services on the VM, but none of the services mentioned are on my agent (https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops)
• Setting a delay before the Azure CLI task gets executed.
• Using Microsoft Hosted agents, which works 100% but is not compliant for my company so not an option.

Pipeline details

trigger:
  branches:
    exclude:
    - master

pool:
  name: SelfHosted-AgentPool
  vmImage: 'windows-latest'

variables:
  environment.name: 'Test'

stages:
- stage: build_and_deploy
  jobs:
  - deployment: VMBackup_Testing  
    displayName: "Enable Backup Protection"
    environment: '$(environment.name)'
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self    
          
         
          - task: PowerShell@2
            inputs:
              filePath: '$(System.DefaultWorkingDirectory)/Templates/Snippets/InstallAzureCLI.ps1'

          - task: AzureCLI@2
            inputs:
              workingDirectory: 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin'
              azureSubscription: 'XXX'
              scriptType: 'ps'
              scriptLocation: 'scriptPath'
              scriptPath: '$(System.DefaultWorkingDirectory)/Templates/Snippets/EnableBackupProtection.ps1'

Install Azure CLI script

# Download and Install Azure CLI
Invoke-WebRequest -Uri https://azcliprod.blob.core.windows.net/msi/azure-cli-2.19.1.msi -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList "/I AzureCLI.msi /quiet"; rm .\AzureCLI.msi

# Update PATH for Powershell to use new installed software
setx /M PATH "$env:Path += ;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin"

# Test if PATH of Azure CLI exists
Test-Path -Path "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin"

# Reload Shell with new PATH 
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")

# Check if AZ CLI is installed
az version

Azure CLI commands script

# Check if script gets executed
Write-Host "Hello, World!"

# AZ CLI commands to enable Backup Protection
az backup protection enable-for-vm `
    --resource-group XXX`
    --vault-name XXXX`
    --vm $(az vm show -g XXX -n XXX --query id) `
    --policy-name DailyBackup

Answer 1

To truly update the PATH for future steps/tasks in the pipeline, try adding:

[Environment]::SetEnvironmentVariable("Path", $env:Path, "Machine")
$azCliPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin"
echo "##vso[task.prependpath]$azCliPath"

to your InstallAzureCLI.ps1.

I don't if both are needed, maybe just echo "##vso[task.prependpath]....

Documentation about updating the path: https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#prependpath-prepend-a-path-to-the--path-environment-variable

Answer 2

Why do you need to install the Azure CLI every time when you run the pipeline on the same self-hosted Windows agent?

Unlike Microsoft-hosted agents, you just need to manually install the required tools on the self-hosted agent machine, then you can use them in the pipelines that run on the agent.

1. Login to your Windows machine (local or VM) where the self-hosted agent is installed.

2. Open the web browser to download the MSI installer of the released latest Azure CLI from here.

3. When installing the Azure CLI via the MSI installer, normally install wizard will automatically add this tool to the system environment variable PATH. After the installation completed, you can open "Edit the system environment variables" on the machine to check it. If it is not added to the system environment variable PATH, you can manually add it.

4. After above steps, as the documentation has recommended, restart the agent service or restart the machine so that the installed Azure CLI tool can be listed in the capabilities of the agent in the pool.

With this way, when you set a pipeline to run on this self-hosted agent, you can directly call the Azure CLI and no need any step to installing Azure CLI in the pipeline.