.Net/VS Test runner fails when running in Azure DevOps

Question

I'm running C# unit tests targeting .Net Framework 4.6.2.

Things are working fine locally, and it also works fine build and running all tests from VS2019 installed on the build agent.

But when running a build in Azure DevOps, using either hosted or self hosted agent, test step fails even though all tests are successful.

Running vstest.console.exe manually from command line, on the build agent, works fine and succeeds. I run this command with the same parameters as done by Azure DevOps.

Error produced from the log (hundreds of them):

##[error][xUnit.net 00:00:00.0457701] Microsoft.VisualStudio.QualityTools.UnitTestFramework: Catastrophic failure: System.TypeInitializationException: The type initializer for 'Xunit.DiaSession' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Reflection.TypeExtensions, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

##[error] at Xunit.DiaSession..cctor() ##[error] --- End of inner exception stack trace --- ##[error] at Xunit.DiaSession..ctor(String assemblyFileName) ##[error] at Xunit.DiaSessionWrapper..ctor(String assemblyFilename) in C:\projects\xunit\src\xunit.runner.utility\Utility\DiaSessionWrapper_DotNet.cs:line 16

##[error] at Xunit.XunitFrontController..ctor(AppDomainSupport appDomainSupport, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink) in C:\projects\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:line 64

##[error] at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo) in C:\projects\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:line 458

Build summary

Total tests: 3679 Passed: 3675 Skipped: 4 Total time: 3.4993 Minutes Vstest.console.exe exited with code 1.

At the top of the log, during initialization of test step:

Multiple versions of same extension found. Selecting the highest version. Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter : 14.0.2505.1 xunit.runner.visualstudio.dotnetcore.testadapter : 99.99.99.0 xunit.runner.visualstudio.testadapter : 99.99.99.0

Pipeline

pool:
  name: Dedicated VS2017
  demands:
  - msbuild
  - visualstudio
  - vstest

steps:
- task: gittools.gitversion.gitversion-task.GitVersion@3
  displayName: GitVersion
  inputs:
    updateAssemblyInfo: true

- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.3.0'

- task: NuGetCommand@2
  displayName: 'NuGet restore Shamrock.Web'
  inputs:
    restoreSolution: '$(Parameters.solution)'

- task: Npm@1
  displayName: 'npm install Documenta'
  inputs:
    workingDir: src/Documenta/Documenta.Web/ClientApp
    verbose: false
  enabled: false

- task: Npm@1
  displayName: 'npm build Documenta/ClientApp'
  inputs:
    command: custom
    workingDir: src/Documenta/Documenta.Web/ClientApp
    verbose: false
    customCommand: 'run build'
  enabled: false

- task: Npm@1
  displayName: 'npm install Shamrock.Web/Angular'
  inputs:
    workingDir: src/Shamrock.Web/Shamrock.Web/Angular
    verbose: false
  enabled: false

- task: Npm@1
  displayName: 'npm build Shamrock.Web/Angular'
  inputs:
    command: custom
    workingDir: src/Shamrock.Web/Shamrock.Web/Angular
    verbose: false
    customCommand: 'run build'
  enabled: false

- task: VSBuild@1
  displayName: 'Build solution'
  inputs:
    solution: src/Shamrock.Web/Shamrock.Web.sln
    msbuildArgs: '/p:SEPrecompilerIncludeRazor=true /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true
    maximumCpuCount: true

- task: VSBuild@1
  displayName: 'Build solution D365'
  inputs:
    solution: src/Shamrock.D365/Shamrock.D365.sln
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    maximumCpuCount: true
  enabled: false

- task: VSBuild@1
  displayName: 'Build solution Documenta'
  inputs:
    solution: src/Documenta/Documenta.sln
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    maximumCpuCount: true
  enabled: false

- task: VSTest@2
  displayName: 'Test Assemblies'
  inputs:
    testAssemblyVer2: |
     **\$(BuildConfiguration)\*tests.dll
     !**\obj\**
     !**\*testadapter.dll
    vsTestVersion: 16.0
    runInParallel: false
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: True

- task: CopyFiles@2
  displayName: 'Copy DACPAC'
  inputs:
    Contents: '**\*.dacpac'
    TargetFolder: '$(build.artifactstagingdirectory)'

- task: CopyFiles@2
  displayName: 'Copy Database Publish Profile'
  inputs:
    Contents: '**\Shamrock.Database.publish.xml'
    TargetFolder: '$(build.artifactstagingdirectory)'

- task: PublishSymbols@1
  displayName: 'Publish symbols path'
  inputs:
    SearchPattern: '**\bin\**\*.pdb'
  enabled: false
  continueOnError: true

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
    ArtifactName: '$(Parameters.ArtifactName)'

Answer 1

The problem occurs only when the "Collect advanced diagnostics in case of catastrophic failures" flag is set to true.

Make sure that it is off, then it works fine.

Known issue now fixed: https://developercommunity.visualstudio.com/content/problem/763234/e0434352clr-issues.html

---

Update 7th January, 2020

This issue showed up again, even though we had aforementioned flag set to false.

The solution we found this time was to add a Tools installer step to the pipeline, and use that from the test step.

Tools installer step in UI

Using the correct test platform version