CODEWITHSUNDEEP
azure-devopsspecflowazure-pipelinesazure-pipelines-release-pipeline
jessehouwing
jessehouwing

Reputation: 114651

Executing SpecFlow tests in VSTS Release phase using SpecRun

We've built a number of tests using SpecFlow and are using SpecRun to run the tests locally and on the build agent. Wed like to take the tests out of the build and stick them in the first release phase of the VSTS Release we have linked to the build.

The team followed the guidance from Tarun and Utkarsh to setup SpecRun in the build, but when we use the Publish Artefact task to package up all the test assemblies, they aren't found when running on the Release Agent.

Upvotes: 2

Views: 1263

Answers (1)

jessehouwing
jessehouwing

Reputation: 114651

When building the project on the build agent, the NuGet references of the solution are restored to the packages folder, this includes the SpecRun.Runner package which contains the Visual Studio Test Adapter to discover and run the tests.

This test runner package isn't copied to the output directory of the Test Project. The VsTest task in the build will automatically detect the test runner from the packages directory, which is why it works like magic in the build phase.

To make it work in Release we've found a few options:

  1. ugly use the publish artefact option and publish the solution packages folder. In the release stage point the VsTest step to the other artefact location to find the Test Adapter: enter image description here

  2. better use the nuget to install the SpecRun.Runner nuget as part of the release stage. This runs the risk of the build downloading a newer or unexpected version of the SpecRun.Runner package.

  3. best check in a separate packages.config with the reference to the SpecRun.Runner project and its dependencies, restore it during the build and attach it as a separate artefact to the build. This ensures the Release stages will be able to use the same version of the SpecRun.Runner and that the version is stored in source control.

Upvotes: 3

Related Questions