azure devops: running unit tests in parallel

Question

We have more than 20 assemblies for which we need to run Unit-Tests by bucketing them in 2 groups which run on 2 different VMS. Both VMs are independent of each other and run in parallel. But, The tests within the VM run sequentially and take a long time to finish(we have a lot of UTs). Something like this:

- job: UnitTests1
timeoutInMinutes: 50
pool:
  vmImage: $(hostedPool)
steps:
  - template: ./templates/run-unittests.yml
    parameters:
      artifactName: $(buildArtifact)
      testAssemblies: ['assembly1', 'assembly2', 'assembly3', 'assembly4', 'assembly5', 'assembly6', 'assembly7', 'assembly8', 'assembly9', 'assembly10']
      runName: UnitTest1

- job: UnitTests2
timeoutInMinutes: 35
pool:
  vmImage: $(hostedPool)
steps:
  - template: ./templates/run-unittests.yml
    parameters:
      artifactName: $(buildArtifact)
      testAssemblies: ['assembly11', 'assembly12', 'assembly13', 'assembly14', 'assembly15', 'assembly16', 'assembly17', 'assembly18', 'assembly19', 'assembly20']
      runName: unitTest2

run-unittests.yml

- ${{ each testAssembly in parameters.testAssemblies }}:
- task: VSTest@2
displayName: 'UT: ${{ testAssembly }}'
inputs:
  vsTestVersion: toolsInstaller
  testAssemblyVer2: |
    $(Pipeline.Workspace)\${{ parameters.artifactName }}\**\MyProject.${{ testAssembly }}.Tests.dll
  runSettingsFile: build/default.runsettings
  runInParallel: false                                       
  runOnlyImpactedTests: false                                 # Test Impact Analysis was evaluated and returns incorrect results
  codeCoverageEnabled: true                                   # collect code coverage
  platform: '$(buildPlatform)'
  configuration: '$(buildConfiguration)'

default.runsettings for test

<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<TargetPlatform>x64</TargetPlatform>
<ParallelizeAssemblies>false</ParallelizeAssemblies>
<ParallelizeTestCollections>false</ParallelizeTestCollections>

How can I start multiple threads/processes to run these UTs in parallel with in 1 VM. runInParallel doesnt seem to do anything.

Answer 1

runInParallel argument means running tests in parallel on multi-core machines. If set, tests will run in parallel leveraging available cores of the machine. This will override the MaxCpuCount if specified in your runsettings file. Click here to learn more about how tests are run in parallel. So you need to check whether you VM is multi-core, and check whether you have specified MaxCpuCount in your runsettings file.

You can also refer to the following documentation to use Visual Studio Test (VSTest) task to run tests in parallel across multiple agents:

https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops#massively-parallel-testing-by-combining-parallel-pipeline-jobs-with-parallel-test-execution