MSTest / VSTest retry (rerun) logic

Question

Unfortunately, there's no native test rerun logic for MStest / VStest

I'm trying to implement a custom logic like this:

Test part:

    static int testNum = 1;

    [TestMethod]
    public void RerunTestOnce_Test()
    {
        testNum = testNum + 1;
        Console.WriteLine("Test started");
        Assert.IsTrue(testNum == 3, $"Test Failed with number {testNum}");

    }

This test should fail for the first time, and pass for the second, when testNum reach value 3.

UP: This is a synthetic example to emulate fail at first run. Real tests are complex and have UI search methods and other work with a system and network, and there's no confidence that everything will be fine during a big & long test suite.

There is a special method for this - RerunTestOnce(), called in TestCleanup:

    [TestCleanup]
    public void TestCleanup()
    {
        TestHelper.RerunTestOnce(TestContext, this);
    }

And here is implementation of RerunTestOnce in test helper class. In it, using Reflection & TestContext we get names of test method and initialize method, and run them again:

 public static void RerunTestOnce(TestContext testContext, object testInstance)
    {
        if (testContext.CurrentTestOutcome == UnitTestOutcome.Failed)
        {
            var type = testInstance.GetType();
            if (type != null)
            {
                var testMethod = type.GetMethod(testContext.TestName);
                var initMethod = type.GetMethods().SingleOrDefault(m=>m.CustomAttributes.SingleOrDefault(a=>a.AttributeType.Name == "TestInitializeAttribute")!= null);
                var cleanupMethod = type.GetMethods().SingleOrDefault(m => m.CustomAttributes.SingleOrDefault(a => a.AttributeType.Name == "TestCleanupAttribute") != null);

                Console.WriteLine($"[WARNING] Method [{testMethod}] was failed in first attempt. Trying to rerun...");
                try
                {
                    initMethod.Invoke(testInstance, null);
                    testMethod.Invoke(testInstance, null);
                }
                catch
                {
                    Console.WriteLine($"[ERROR] Method [{testMethod}] was failed in second attempt. Rerun finished.");
                }
            }
        }
    }

Everything is ok, on the second attempt test method passes, but in the end I see Failed result and assert error message from first attempt:

Test Failed - RerunTestOnce_Test
Message: Assert.IsTrue failed. Test Failed with number 2

How and when MSTest creates test result - is it possible to update test result after second attempt to last result?

Answer 1

Update

Today I learned that you can actually write your own TestMethod Attribute.

The default implementation looks like this

public class TestMethodAttribute : Attribute
{
    public virtual TestResult[] Execute(ITestMethod testMethod)
    {
        return new TestResult[1] { testMethod.Invoke(null) };
    }
}

So you actually can create your own TestMethodWithRetry attribute and use that on a method

[TestMethodWithRetry]
public void TestRetry()
{
    var x = new Random().Next(0, 2);
    Assert.AreEqual(1, x);
}

// or
[TestMethodWithRetry(Count = 10)]
public void TestRetry()
{
    var x = new Random().Next(0, 2);
    Assert.AreEqual(1, x);
}

// even works with DataRow
[TestMethodWithRetry(Count = 10)]
[DataRow(2)]
[DataRow(5)]
[DataRow(10)]
public void TestRetry(int max)
{
    var x = new Random().Next(0, max);
    Assert.AreEqual(1, x);
}

public class TestMethodWithRetryAttribute : TestMethodAttribute
{

    public int Count { get; set; } = 1;

    public override TestResult[] Execute(ITestMethod testMethod)
    {
        var count = Count;
        TestResult[] result = null;
        while (count > 0)
        {
            try
            {
                result = base.Execute(testMethod);
                if (result[0].TestFailureException != null)
                {
                    throw result[0].TestFailureException;
                }
            }
            catch (Exception) when (count > 0)
            {
            }
            finally
            {
                count--;
            }
        }
        return result;
    }

}

This is just a simple implementation but it seems to work surprisingly well.

---

I came up with the following solution

public static void Retry(Action test, int retry = 10, int sleep = 0, [CallerMemberName] string testName = null)
{
    int current = 1;
    retry = Math.Max(1, Math.Min(retry, 10));
    while (current <= retry)
    {
        try
        {
            test();
            break;
        }
        catch (Exception ex) when (current < retry)
        {
            Debug.WriteLine("Test {0} failed ({1}. try): {2}", testName, current, ex);
        }

        if (sleep > 0)
        {
            Thread.Sleep(sleep);
        }
        current++;
    }
}

usage

[TestMethod]
public void CanRollbackTransaction()
{
    Helpers.Retry(() =>
    {
        var even = DateTime.Now.Second % 2 == 0;
        Assert.IsTrue(even);
    }, 3, 1000);
}

Answer 2

The MSTest Test Framework by itself does not support a native test-rerun logic.
Please consider using MSTestEx a set of extensions to the MSTest Test Framework, that does support test-rerun logic: https://www.nuget.org/packages/MSTestEx/