CODEWITHSUNDEEP
azureazure-devopsazure-pipelinesazure-cliazure-cli2
Jason
Jason

Reputation: 2673

Why is running an Azure CLI command in a pipeline super slow?

I am very new to ADO pipelines in Azure. I have a situation where I have the build ID of a pipeline run that created an artifact. I need to download this artifact in a different pipeline. In order to use DownloadPipelineArtifact@2 you have to provide both the pipeline an buildId. the docs

In my case, I only have the buildId and figured I could look up the pipeline via an az command, like so:

- task: AzureCLI@2
    displayName: lookup pipeline ID
    inputs:
      azureSubscription: $(ServicePrincipal)
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        pipelineId=$(az pipelines runs show --id ${{ parameters.buildId }} --query "definition.id")
        echo build ${{ parameters.buildId }} belongs to pipeline $pipelineId
        echo "##vso[task.setvariable variable=pipelineId]$pipelineId"
    env:
      AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

This all works as expected, BUT when I run that same az command from my local machine, it returns in < 1 second... When it runs in my pipeline, it takes closer to 1 minute.

enter image description here

what am I doing wrong (if anything) and what could I do to make this command happen faster?

UPDATE 1

Based on jessehouwing's answer, I updated my step like so (I'm on a Windows-hosted agent, not self-hosted):

parameters:
  buildId: ''

steps:
- bash: |
    pipelineId=$(az pipelines runs show --id $BUILD_ID --query "definition.id")
    echo build $BUILD_ID belongs to pipeline $pipelineId
    echo "##vso[task.setvariable variable=pipelineId]$pipelineId"
  displayName: lookup pipeline ID
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
    BUILD_ID: ${{ parameters.buildId }}

And it actually ran SLOWER,

enter image description here

enter image description here

So, while using bash still got me the data I needed, it didn't speed anything up.

UPDATE 2

so, I updated my lookup-pipeline step as you suggested, but it had a typo....and the pipeline ID was never returned:

enter image description here

much to my surprise, the download step did not fail.... it actually downloaded the correct artifact. I tested a redeploy of an older build just to make sure it wasn't grabbing the "latest" build. Sure enough, from what I can tell....despite the docs saying otherwise, the pipeline doesn't appear to be required when attempting to download an artifact from a pipeline in the same org. Therefore, I have eliminated the lookup step entirely and the entire pipeline works as expected.

Another tragic case of out of date MS docs

Upvotes: 0

Views: 2520

Answers (2)

JustMe
JustMe

Reputation: 208

I suggest you to switch to macos-latest and use PowerShell Core

Upvotes: 0

jessehouwing
jessehouwing

Reputation: 114751

There is a bug in the current hosted runner image which causes az cli to discover all of the installed modules and extensions on the first call, because this data is lost in the provisioning process. This enumeration takes around 50s.

Update: A number of pull-requests are making their way to the public runners, the ones in green have already been deployed.

  • ✅ actions/runner-images/8410 - Warms up az devops, installs Python's keychain package and run az devops warmup on Windows
  • ✅ actions/runner-images/8410 - Redirects all temporary/cache files to a central folder, runs modules during warmup on Windows.
  • 🔁 actions/runner-images/8441 - Redirects all temporary files to a central folder, installs Python's keychain package and runs modules during warmup on Ubuntu.
  • 🔁 azure/azure-cli/27545 - Fixes case-sensitive check against the command index and prevents unwanted index invalidation.
  • ✅ actions/runner-images/8388 - Disables Windows Storage Service.
  • ✅ actions/runner-images/8431 - Disables Windows Update & Windows Update Medic services

With these changes, first-call performance is now less than ~15s instead of ~60s or more on Windows.

I've captured all of my findings from the last week looking into this issue in an extensive blog article.

Upvotes: 5

Related Questions