CODEWITHSUNDEEP
tfstfsbuildmspec
Tedford
Tedford

Reputation: 2932

Test category filter in a TFS 2015 vNext build with the mspec test adapter not filtering tests

I have code base in which the unit tests are written with Machine Specifications leveraging the nuget based test runner Machine.Specifications.Runner.VisualStudio, v2.10 to execute the tests. It works fine from the context of Visual Studio (2015 & 2017) and filtering by Trait works as expected. However, when using the Test Assemblies build step it does not seem to honor the category filter. Is there something special with how the TFS build agent runs the test adapter compared to visual studio?

Example Test

    [Subject(typeof(RetrieveConfiguration))]
    [Tags(Categories.Manual)]
    public class When_hitting_the_general_services_uri : SpecificationContext
    {
        private static TestResult result;

        Establish context = () =>
        {
            IServiceInfo serviceInfo = Substitute.For<IServiceInfo>();
            serviceInfo.Url = "";
            environment.GetService("Confiugration", "Retrieve").Returns(serviceInfo);
            x509Manager.LoadFromSignature(ValidSignature).Returns(LoadFromMachine(ValidSignature));
        };

        Because of = () => error = Catch.Exception(() => result = sut.Execute(new Uri("https://myproductionuri/retrieve"), environment));

        It should_have_the_succeeded = () => result.Result.ShouldEqual(StepResult.Success);
    }

Build Step Configuration TFS vNext build test step

Build Log

...
2017-08-10T20:49:44.8717852Z ##[debug]Calling Invoke-VSTest for all test assemblies
2017-08-10T20:49:44.9655216Z Working folder: E:\B39\BA78\WS\18
2017-08-10T20:49:44.9655216Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll"  /TestCaseFilter:"TestCategory=ContinuousIntegration" /EnableCodeCoverage /logger:trx /TestAdapterPath:"E:\B39\BA78\WS\18\s"
2017-08-10T20:49:45.1999042Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-08-10T20:49:45.1999042Z Copyright (c) Microsoft Corporation.  All rights reserved.
2017-08-10T20:49:45.5592884Z Starting test execution, please wait...
2017-08-10T20:49:56.8721150Z Information: Machine Specifications Visual Studio Test Adapter - Executing tests in E:\B39\BA78\WS\18\s\Src\Test\Verifier.Reporting.Azure.Storage.Spec\bin\Release\Verifier.Reporting.Azure.Store.Spec.dll
2017-08-10T20:50:01.5285749Z Passed   Verifier.Reporting.Azure.Store.Spec.When_publishing_a_report.should_have_succeeded
...

Update 8/25 - added the requested screen shots and feedback

Test Explorer without filtering Viewing tests in VS Test Explorer without any filtering

Notice there are 16 total tests, the indicates ones starting with when hitting are integration tests which are not expected to run within the context of the build agent.

Test Explorer with filtering on Category Filtering test using category in VS Test Explorer

The total number of tests has decreased from 16 to 14. Since the test did not have the requested tag it was dropped from the test run.

Running vs2015 vstest.console.exe Screen shot of running vstest.console.exe on dev machine

As for running the test outside of visual studio, it would appear that the test runner is experiencing issues loading the test adapter on my dev machine, whereas the adapter runs fine in Visual Studio and on the build agent.

Upvotes: 2

Views: 1366

Answers (1)

PatrickLu-MSFT
PatrickLu-MSFT

Reputation: 51073

The vstest task is just using vstest.console.exe to execute tests. The test Filter Criteria in TFS VStest task works the same way as the console option /TestCaseFilter of vstest.console.exe. 

"TestCategory=ContinuousIntegration"

By above, I did not see such TestCategory name in your code, if we run the test by using command line(vstest.console.exe), we have to specify the matched name, which means at least we name a TestCategory attribute above your test method code i.e, my test method code :

[TestCategory("nine"), TestMethod()]
    public void TestMethod1()
    {
        Assert.AreEqual(1, 1);
    }

I need run it by using code below in command line:

Vstest.console.exe UnitTestvstsada.dll /TestCaseFilter:TestCategory=nine

It will filter the test successfully and will get the same result in TFS build.

As for filter in Test Explorer, there is no such option to filter the test. Only a configure continuous integration which actually is not a filter. If you don't mind, please kindly show how did you using the filter trait:ContinuousIntegration in the test of test run via visual studio.

enter image description here

Afraid using the filter trait:ContinuousIntegration that is not equivalent to TestCategory=ContinuousIntegration in the TFS2015 build agent.

Upvotes: 0

Related Questions