Azure release pipeline to Publish Test Results without rerunning tests

Question

I'm trying to create a release pipeline that allows me to automatically update the test results in a Test plan. I run the tests through nunit-console-runner in a build pipeline. I have already successfully built a build pipeline that runs tests and reads off results. This is the YAML for that pipeline:

pool:
  name: YourAgentPoolName
  demands:
  - msbuild
  - visualstudio
  - vstest

variables:
  BuildConfiguration: 'Release'
  BuildPlatform: 'any cpu'

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 7.0.x'
  inputs:
    version: 7.0.x

- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'

- task: VSBuild@1
  displayName: 'Build solution **\*.sln'
  inputs:
    msbuildArgs: '/p:Configuration=$(BuildConfiguration) /p:Platform="$(BuildPlatform)" /p:OutputPath=$(Build.ArtifactStagingDirectory)'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true

- script: |
   set PATH=%PATH%;C:\ProgramData\chocolatey\bin
         choco install nunit-console-runner -y
  displayName: 'Install nunit-console-runner'

- script: |
       if exist "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" (
         echo ✅ NUnit is installed correctly.
       ) else (
         echo ❌ NUnit is NOT installed! Check the installation.
         exit 1
       )
  displayName: 'Verify NUnit Installation'

- script: |
   echo "Checking for test DLL in build output..."
   dir $(Build.ArtifactStagingDirectory) /s /b | findstr "YourTestProject.dll"
  displayName: 'Verify DLL Location'

- script: |
   cd "C:\Program Files (x86)\NUnit.org\nunit-console"
   nunit3-console.exe $(Build.ArtifactStagingDirectory)\YourTestProject.dll --where "$(FeaturesToTest)"
  displayName: 'Run NUnit Tests'
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(Build.SourcesDirectory)'
    mergeTestResults: true

- task: PublishPipelineArtifact@1
  displayName: 'Publish Pipeline Artifact'
  inputs:
    targetPath: '$(Build.SourcesDirectory)'
    artifact: XmlLogsArtifact

I want to make a release pipeline that makes it where you can go to a Test Plan, click on the tests you want to update the status of, click

Execute > Run With Options > Automated tests using release stage

to update the results. This is the YAML for each task in my release pipeline so far.

steps:
- task: DownloadPipelineArtifact@2
  displayName: 'Download Test Artifacts'
  inputs:
    artifactName: XmlLogsArtifact
    targetPath: '$(System.ArtifactsDirectory)'
steps:
- task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
  displayName: 'What are certain things'
  inputs:
    Script: |
     Write-Host "System.ArtifactsDirectory: $(System.ArtifactsDirectory)"
     Write-Host "Agent.TempDirectory: $(Agent.TempDirectory)"
     Write-Host "System.DefaultWorkingDiretory: $(System.DefaultWorkingDirectory)"
     Get-ChildItem -Path "$(System.ArtifactsDirectory)" -Recurse
steps:
- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Visual Studio Test Platform Installer'
steps:
- task: VSTest@2
  displayName: 'VsTest - testAssemblies'
  inputs:
    testAssemblyVer2: '**/Trintech.Tests.Cadency.dll'
    searchFolder: '$(Build.ArtifactStagingDirectory)'
    resultsFolder: TestResult.xml
    runOnlyImpactedTests: false
    runTestsInIsolation: true
  continueOnError: true
steps:
- task: PublishTestResults@2
  displayName: 'Publish Test Results **/TestResult.xml'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(System.ArtifactsDirectory)'
    mergeTestResults: true
    failTaskOnMissingResultsFile: true

I've had the release pipeline deploy successfully, but it will fail at the "Validating stage" when it runs. The error it currently gives is "The selected stage does not have the right version or settings of the Visual Studio Test task to run tests."

Do I need the VsTest task at all? I thought no, because I don't want to rerun the same tests twice. But the only time I've had this work was when I was doing that. When I remove that task, I get the same error. Please help.

Answer 1

If your expectation is to Run Automated tests from test plans, then Yes, you need to use the VSTest task in the environment(stage) of your release pipeline.

The functionality to Run with options of Automated tests using release stage will validates the selected stage to ensure the Visual Studio Test task is present and has valid settings. The property testSelector: testRun of the VSTest is one of those required settings.

As another prerequisite, you also need to Associate automated tests with test cases, so that the VSTest task can first acknowledge the test case(s) in the test plan that you intend to run and then find the corresponding associated build assembly/assemblies downloaded as pipeline artifacts into searchFolder.

Regarding your current workflow, if possible, you may consider moving the steps to run tests from the build pipeline into the VSTest step of the release pipeline to avoid duplicate tests.

Please note that the above feature is still a manually triggered test workflow using release stage. To automatically trigger the release pipeline to pickup tests in your test plan, you can use the 3rd testSelector: testPlan, which also requires you to associate test assemblies with test cases in the first place. This release also populates the execution history of each test case in the test plan, whose associated assembly is tested and can be triggered automatically with CD trigger enabled.