CODEWITHSUNDEEP
c#nunit
Agent Shoulder
Agent Shoulder

Reputation: 585

How to retry a NUnit TestCase?

I want to acheive a setup where I can set a desired retry count for each test, so that I can for example, retry all my failing tests once before actually failing them. I've structured my tests this way:

    [TestCase(“Some parameter”, Category = “Test category”, TestName = “Name of test”, Description = “Description of test”)]
    public void SomeTestName(string browser) {
    //Test script
    }

If I had used [Test] instead of [TestCase] I could have just added a [Retry(1)] attribute, but how do I achieve the same behavior with [TestCase]? I've already glanced at NUnit retry dynamic attribute which has a quite neat solution, but unfortunately it has no effect when I try to apply it to a [TestCase]

Upvotes: 3

Views: 7361

Answers (2)

Reza Biglari
Reza Biglari

Reputation: 74

For anyone here who wants to see a tweaked version of @Charlie's solution for TestCaseSource.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true, Inherited = false)]
    public class RetryTestCaseSourceAttribute : TestCaseSourceAttribute, IRepeatTest
    {
        #region constructors
        public RetryTestCaseSourceAttribute(string sourceName) : base(sourceName){}
        public RetryTestCaseSourceAttribute(Type sourceType) : base(sourceType){}
        public RetryTestCaseSourceAttribute(Type sourceType, string sourceName) : base(sourceType, sourceName){}
        public RetryTestCaseSourceAttribute(string sourceName, object[] methodParams) : base(sourceName, methodParams){}
        public RetryTestCaseSourceAttribute(Type sourceType, string sourceName, object[] methodParams) : base(sourceType, sourceName, methodParams){}
        #endregion

        #region repeat components
        public int MaxTries { get; set; }
        TestCommand ICommandWrapper.Wrap(TestCommand command) => new RetryAttribute.RetryCommand(command, MaxTries);
        #endregion
    }

You can then use:

[RetryTestCaseSource(nameof(GetBrowsers), MaxTries = 3)]
public void SomeTestName(string browser)
{
    // Your test code
}
public static IEnumerable<string> GetBrowsers()
{
    return new List<string>() {/* browsers */};
}

This solution is verified for NUnit 3.12.0.

Upvotes: 1

Charlie
Charlie

Reputation: 13746

Per the docs: "RetryAttribute is used on a test method to specify that it should be rerun if it fails, up to a maximum number of times."

That is, the argument is not the number of retries as you might think but the total number of attempts to run the test and [Retry(1)] has no effect at all, no matter where you use it. Since this is a possible point of confusion, I just edited that page to give an explicit warning.

If you were to try to use RetryAttribute on a class, you would get a compiler warning because it can only be used on methods. However, in NUnit, a method may represent either a single test or a group of parameterized tests. In the case of a parameterized test, the attribute currently has no effect.

It would be possible for the NUnit team to decide that this attribute applies to each individual test case and modify nunit accordingly. It would also be possible for TestCaseAttribute to take an optional parameter specifying a retry count. For a long-term solution, you might want to ask them for one or the other of those options.

In the shorter term, as a workaround, you could consider deriving your own attribute from TestCaseAttribute. Here is some (untested) code to get you started...

using System;
using NUnit.Framework.Interfaces;
using NUnit.Framework.Internal.Commands;

namespace NUnit.Framework
{
  [AttributeUsage(AttributeTargets.Method, AllowMultiple = true, Inherited = false)]
  public class RetryTestCaseAttribute : TestCaseAttribute, IRepeatTest
  {
    // You may not need all these constructors, but use at least the first two
    public RetryTestCaseAttribute(params object[] arguments) : base(arguments) { }
    public RetryTestCaseAttribute(object arg) : base(arg) { }
    public RetryTestCaseAttribute(object arg1, object arg2) : base(arg1, arg2) { }
    public RetryTestCaseAttribute(object arg1, object arg2, object arg3) : base(arg1, arg2, arg3) { }

    public int MaxTries { get; set; }

    // Should work, because NUnit only calls through the interface
    // Otherwise, you would delegate to a `new` non-interface `Wrap` method.
    TestCommand ICommandWrapper.Wrap(TestCommand command)
    {
      return new RetryAttribute.RetryCommand(command, MaxTries);
    }
  }
}

You would use this as follows

[RetryTestCase("some parameter", MaxTries=3)]
public void SomeTestName(string browser)
{
  // Your test code
}

Some Notes about the above:

  1. I have compiled this code but haven't tested it. Please post a comment if you try it out, especially if it needs modifications.

  2. The code relies on some knowledge of NUnit internals and could break in the future. A more comprehensive implementation would be needed to make it future-proof. In particular, I used the fact that IRepeatTest is based on ICommandWrapper but adds no methods. I believe that each of the two interfaces is needed where I put them because NUnit examines them at different points in its code.

  3. This code is about three times as many lines of code compared to what would be needed to add a retry count to TestCaseAttribute! Ask the NUnit Project if you would like that feature - or contribute it yourself!

Upvotes: 8

Related Questions