How to stop test if one test fails using vstest.console.exe?

Question

I have a batch file that contains multiple tests defined in a similar form below.

vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t1,t2,t3,t4,t5"

The tests run in order from t1 to t5. However, I want to stop vstest if any one of the tests fails. Is this possible using vstest.console.exe?

Btw, the contents of my test.runsettings is

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <MSTest>
    <ForcedLegacyMode>true</ForcedLegacyMode> 
      <KeepExecutorAliveAfterLegacyRun>true</KeepExecutorAliveAfterLegacyRun> 
  </MSTest>
</RunSettings>

I have checked the Documentation for runsettings, it seems there is no flag/attribute for this case.

Answer 1

In addition to lukbl's answer, you can do the same assembly-wide, so if you have multiple test classes, you'll have a global management on the tests, during vstest.console.exe's run-time (if you're calling it multiple times for example).

Care should be taken depending on how you're employing vstest.console (or mstest). If you're load-balancing between multiple test agents, each test agent will run their own vstest.console.exe, and thus will have their own assembly-level values, so the session management will be limited by the group of tests running on the same agent. Let's say this approach will give you management on the whole set of tests you run with a command: vstest.console.exe /filter: tests.dll

That means regardless of the scope of your session_failed variable (class-wide or assembly-wide) if you end up running different tests from same class with different vstest.console.exe calls, you'll lose the variable value, or the control.

That being said, a simple approach for multi-class test scenario:

[TestClass]
public static class TestSettings
{
    public static bool SessionTestsFailed = false;

    [AssemblyInitialize]
    public static void runsBeforeAnyTest(TestContext t)
    {
        TestSettings.SessionTestsFailed = false;
    }
}

[TestClass]
public class Tests1
{
    public TestContext TestContext { get; set; }

    [TestInitialize()]
    public void MyTestInitialize()
    {
        if (TestSettings.SessionTestsFailed)
            Assert.Fail("Session failed, test aborted");
    }

    [TestCleanup]
    public void MyTestFinalize()
    {
        if (TestContext.CurrentTestOutcome != UnitTestOutcome.Passed)
            TestSettings.SessionTestsFailed = true;
    }

    [TestMethod]
    public void test11()
    {
        Console.WriteLine("test11 ran");
        Assert.Fail("fail the test");
    }

    [TestMethod]
    public void test12()
    {
        Console.WriteLine("test12 ran");
        Assert.Fail("fail the test");
    }
}

[TestClass]
public class Tests2
{
    public TestContext TestContext { get; set; }

    [TestInitialize()]
    public void MyTestInitialize()
    {
        if (TestSettings.SessionTestsFailed)
            Assert.Fail("Session failed, test aborted");
    }

    [TestCleanup]
    public void MyTestFinalize()
    {
        if (TestContext.CurrentTestOutcome != UnitTestOutcome.Passed)
            TestSettings.SessionTestsFailed = true;
    }

    [TestMethod]
    public void test21()
    {
        Console.WriteLine("test21 ran");
        Assert.Fail("fail the test");
    }

    [TestMethod]
    public void test22()
    {
        Console.WriteLine("test22 ran");
        Assert.Fail("fail the test");
    }

And here's an easy way to update all your test initialize methods at once, if their signature is the same, using regexp matching, visual studio replace all: find:

(\s*)public void MyTestInitialize\(\)(\s*)(\r*\n)(\s*){(\r*\n)

replace:

$1public void MyTestInitialize()$3$4{$1\tif (TestSettings.SessionTestsFailed) Assert.Fail("Session failed, test aborted");

and similar for TestFinalize().

Answer 2

If it is an option for you, then you could introduce base class for tests with cleanup, initialize methods and TestContext property.

In cleanup method you will check if test is failed and by triggering Assert.Fail in TestInitialize you don't allow any other test to pass after that.

[TestClass]
public class BaseTest
{
    private static bool _failAllTests;
    public TestContext TestContext { get; set; }

    [TestInitialize]
    public void InitializeMethod()
    {
        if (_failAllTests)
        {
            Assert.Fail("Fail all tests");
        }
    }

    [TestCleanup]
    public void CleanUpMethod()
    {
        if (TestContext.CurrentTestOutcome == UnitTestOutcome.Failed)
        {
            _failAllTests = true;
        }
    }
}
[TestClass]
public class UnitTest1 : BaseTest
{
    [TestMethod]
    public void TestMethod1()
    {
        Assert.Fail("TestMethod1 failed!");
    }
    [TestMethod]
    public void TestMethod2()
    {
        Assert.IsTrue(true, "TestMethod2 passed!");
    }
}

Answer 3

If the number of test to run is small, like in your example, you can split it to multiple runs of vstest.console.exe and check ERRORLEVEL in the batch. If you ERRORLEVEL is not 0, it means a test fails, you can exit the batch.

vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t1"
IF ERRORLEVEL 1 GOTO exit
vstest.console.exe Test.dll /Settings:"test.runsettings" /Tests:"t2"
IF ERRORLEVEL 1 GOTO exit
...

:exit