How to set the test case sequence in xUnit

Question

I have written the xUnit test cases in C#. That test class contains so many methods. I need to run the whole test cases in a sequence. How can I set the test case sequence in xUnit?

Answer 1

In xUnit 2.* this can be achieved using the TestCaseOrderer attribute to designate an ordering strategy, which can be used to reference an attribute that is annotated on each test to denote an order.

For example:

Ordering Strategy

[assembly: CollectionBehavior(DisableTestParallelization = true)] 

public class PriorityOrderer : ITestCaseOrderer
{
    public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
    {
        var sortedMethods = new SortedDictionary<int, List<TTestCase>>();

        foreach (TTestCase testCase in testCases)
        {
            int priority = 0;

            foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
                priority = attr.GetNamedArgument<int>("Priority");

            GetOrCreate(sortedMethods, priority).Add(testCase);
        }

        foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
        {
            list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
            foreach (TTestCase testCase in list)
                yield return testCase;
        }
    }

    static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key) where TValue : new()
    {
        TValue result;

        if (dictionary.TryGetValue(key, out result)) return result;

        result = new TValue();
        dictionary[key] = result;

        return result;
    }
}

Attribute

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
    public TestPriorityAttribute(int priority)
    {
        Priority = priority;
    }

    public int Priority { get; private set; }
}

Test Cases

[TestCaseOrderer("FullNameOfOrderStrategyHere", "OrderStrategyAssemblyName")]
public class PriorityOrderExamples
{
    [Fact, TestPriority(5)]
    public void Test3()
    {
        // called third
    }

    [Fact, TestPriority(0)]
    public void Test2()
    {
      // called second
    }

    [Fact, TestPriority(-5)]
    public void Test1()
    {
       // called first
    }

}

xUnit 2.* ordering samples here

Answer 2

I was using xUnit V3 so none of the existing code/packages worked, but writing one was simple enough:

public class TestOrderer : ITestCaseOrderer
{
    public IReadOnlyCollection<TTestCase> OrderTestCases<TTestCase>(IReadOnlyCollection<TTestCase> testCases) where TTestCase : notnull, ITestCase
    {
        return testCases.OrderBy(t => GetTestMethodOrder(t)).CastOrToReadOnlyCollection();
    }

    private static int GetTestMethodOrder<TTestCase>(TTestCase testCase) where TTestCase : notnull, ITestCase
    {
        var testMethod = testCase.TestMethod as XunitTestMethod ?? throw new InvalidOperationException("Error trying to figure out test method order.");
        return testMethod.Method.GetCustomAttribute<OrderAttribute>()?.Order ?? throw new InvalidOperationException("Error trying to figure out test method order.");
    }
}

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class OrderAttribute(int order) : Attribute
{
    public int Order { get; private set; } = order;
}

Usage:

[Fact, Order(1)]
public void Test1()
{
}

[Fact, Order(2)]
public void Test2()
{
}

Answer 3

For some reason, XUnit.Priority didn't work for me. In my test cases, it wasn't running the tests in the priority order specified.

So I tried XUnitPriorityOrderer, which is similar to use but was working (To quickly test it, save the following code in a text editor as OrderedXUnitTests.linq, then open it with LinqPad 6 and execute it. Alternatively, you can also copy the TestClass to Visual Studio and add XUnit, XUnit.Runner.VisualStudio and XUnitPriorityOrderer):

<Query Kind="Program">
  <NuGetReference>XUnitPriorityOrderer</NuGetReference>
  <Namespace>Xunit</Namespace>
  <Namespace>XUnitPriorityOrderer</Namespace>
</Query>


#load "xunit"

// using XUnitPriorityOrderer
// see: https://github.com/frederic-prusse/XUnitPriorityOrderer
void Main()
{
    RunTests();  // Call RunTests() or press Alt+Shift+T to initiate testing.
}


#region private::Tests

[TestCaseOrderer(CasePriorityOrderer.TypeName, CasePriorityOrderer.AssembyName)]
public class TestClass
{
    static List<string> Order { get; set; }
    
    public TestClass()
    {
        Order = Order ?? new List<string>();
    }

    [Fact, Order(2)]
    void Test_Xunit_AnotherTest()
    {
        Order.Add("Test_Xunit_AnotherTest");
        Assert.True(3 + 1 == 4);
    }

    [Fact, Order(1)]
    void Test_Xunit()
    {
        Order.Add("Test_XUnit");
        Assert.True(1 + 1 == 2);
    }

    [Fact, Order(99)]
    void Print_Order()
    {
        Order.Add("Print_Order");
        var strOrder = string.Join(", ", Order.ToArray());
        strOrder.Dump("Execution Order");
        Assert.True(true);
    }
}
#endregion

This will run the tests in given order (Order(1), Order(2) and then Order(99)) and will dump the executed tests finally (test method Print_Order()).

Answer 4

If you really have the need to prioritize your tests (probably not your unit tests) you can use Xunit.Priority. I have used it for some integration testing and works really well and simple without the overhead of having to write your prioritization classes, for simple case scenarios

Answer 5

Testpriority: at the bottom of this page.

[PrioritizedFixture]
public class MyTests
{
    [Fact, TestPriority(1)]
    public void FirstTest()
    {
        // Test code here is always run first
    }
    [Fact, TestPriority(2)]
    public void SeccondTest()
    {
        // Test code here is run second
    }
}

BTW, I have the same problem right now. And yes, it is not the clean art.. but QA wanted a manual test.. so an automated test with a specific order already is a big leap for them.. (cough) and yes, it is not really unit testing..

Answer 6

You can't, by design. It's deliberately random in order to prevent anyone getting one of those either by desire or by accident.

The randomness is only for a given Test class, so you may be able to achieve your goals by wrapping items you want to control the order of inside a nested class - but in that case, you'll still end up with random order whenever you have more than two Test Methods in a class.

If you're trying to manage the building up of fixtures or context, the built-in IUseFixture<T> mechanism may be appropriate. See the xUnit Cheat Sheet for examples.

But you really need to tell us more about what you're trying to do or we'll just have to get speculative.