CODEWITHSUNDEEP
xunit
metdos
metdos

Reputation: 13929

How do I skip specific tests in xUnit based on current platform

However, I have found that while 400 of these tests can run (in order), that certain tests either hang the xUnit runner, or bring it down entirely.

I don't care if certain tests are not able to run on Linux, certain tests are to do with the DTC and some unmanaged gumph that we don't need to support there.

What I do want however, is to apply an ignore to those tests, and have the fact that the test was ignored flagged properly in the build output.

The question can be boiled down to I guess a number of possible solutions

If I could avoid modifying the original code too much that would be grand, as the code isn't really mine to change, and applying lots of cross-platform hacks probably won't go down too well.

Upvotes: 183

Views: 176410

Answers (10)

Dominik Lemberger
Dominik Lemberger

Reputation: 2426

There is a new options now.

Add Nuget Package Xunit.SkippableFact, which allows you to use [SkippableFact] instead of [Fact] (or [SkippableTheory] ) and you can use Skip.<xyz> within a Tests to dynamically Skip the Test during runtime.

Example:

[SkippableFact]
public void SomeTestForWindowsOnly()
{
    Skip.IfNot(Environment.IsWindows);

    // Test Windows only functionality.
}

Upvotes: 77

Luke Vo
Luke Vo

Reputation: 20678

With xUnit v3, you can skip it without external packages:

    [Fact]
    public void SkippableTest()
    {
        Assert.Skip("This test is skipped");

        var isWindows = RuntimeInformation.IsOSPlatform(OSPlatform.Windows);
        Assert.SkipWhen(isWindows, "Skipping because running Windows");
        Assert.SkipUnless(isWindows, "Skipping because it is not running on Windows");
    }

From What's News:

Dynamically skippable tests We have added the ability to dynamically skip tests via the [Fact] and [Theory] attributes, in addition to the Assert.Skip family of assertions mentioned above.

The ability to dynamically skip tests has been added to support determining when to skip a test at runtime rather than at compile time. An example of a reason to dynamically skip a test at runtime would include: skipping a test based on the execution environment. That might include the operating system (Windows vs. macOS vs. Linux) or the runtime environment (interactive tests run by developers vs. tests run during continuous integration). The project template xunit3-extension illustrates one way to add a dynamic skip system (via an attribute) based on the operating system.

You may set either SkipUnless or SkipWhen (but not both) to point at a public static property on the test class which returns bool; to skip the test, return false for SkipUnless or true for SkipWhen. You may also place the public static property on another class by setting SkipType. Note that setting both SkipUnless and SkipWhen will result in a runtime test failure.

During discovery, tests which have Skip set along with one of SkipUnless or SkipWhen will not show up as statically skipped tests; instead, the property value will be inspected at runtime during test execution to determine if it should be skipped at runtime. This means that, unlike statically skipped tests, the entire test pipeline is run for such tests, including BeforeAfterTestAttribute instances, test class creation, and test class disposal.

Upvotes: 0

JReis
JReis

Reputation: 99

Using Fact Attribute works for me,

public class FactSkipTest : FactAttribute
{
    private readonly string _reason = "Test skipped: Requires elevated privileges (Administrator mode) for execution."";

    public override string Skip => skipIfIsNotAdministrator() ? _reason : null;

    public bool skipIfIsNotAdministrator()
    {
        var identity = WindowsIdentity.GetCurrent();
        var principal = new WindowsPrincipal(identity);
        return !principal.IsInRole(WindowsBuiltInRole.Administrator);
    }
}

public class AdministrationRightsTests
{
    [FactSkipTest]
    void checkThatApplicationDoesNotExists()
   { 
     //some tests
   }
}

Upvotes: 1

AnGG
AnGG

Reputation: 831

You can use this custom attribute

using System;
using System.Runtime.InteropServices;
using Xunit;

[AttributeUsage(AttributeTargets.Method, Inherited = false, AllowMultiple = false)]
public class OsFactAttribute : FactAttribute
{
    public OsFactAttribute(params string[] allowedPlatforms)
    {
        try
        {
            if (!Array.Exists(allowedPlatforms, platform => RuntimeInformation.IsOSPlatform(OSPlatform.Create(platform))))
                Skip = $"Skip, this is allowed only on '{string.Join(", ", allowedPlatforms)}' platforms";
        }
        catch { /* this will not be skipped */ }
    }
}

Like this, with one or more OS names

using System;
using System.Runtime.InteropServices;
using Xunit;


[OsFact(nameof(OSPlatform.Windows), nameof(OSPlatform.Linux))]
public void Get_Input_ShouldBeEqual()
{

}

The same can be done as well for TheoryAttribute

See also:

https://josephwoodward.co.uk/2019/01/skipping-xunit-tests-based-on-runtime-conditions

Upvotes: 2

dpnmn
dpnmn

Reputation: 543

To add to the previous answers regarding SkippableFact: Note that each of the tests are still constructed - the constructor is run.

If you have timeconsuming code in a base class constructor, an alternative is to gather environment-specific test cases in suitable files and run the environment check in the constructor:

        if (!SupportsTemporalQueries())
            throw new SkipException("This test class only runs in environments support temporal queries");

This can speed up the test run considerable. In our system we either extend a "generic" base test class (runs in all environments) or an environment-specific base test class. I find this easier to maintain than filtering in pipelines or other solutions.

Upvotes: 0

Ramin Bateni
Ramin Bateni

Reputation: 17415

The Dominik's solution work for me by this code:

[SkippableFact]
public void Get_WhenCall_ReturnsString()
{
    // Arrange
    Skip.IfNot(RuntimeInformation.IsOSPlatform(OSPlatform.Windows));

    // Act

    // Assert

}

Upvotes: 3

wickdninja
wickdninja

Reputation: 1039

[Fact(Skip="reason")]

works but I prefer to use traits

[Fact, Trait("type","unit")]
public void MyUnitTest(){
  // given 
  // when
  // then
}

[Fact, Trait("type","http")]
public void MyHttpIntegrationTest(){
  // given 
  // when do things over HTTP
  // then
}

usage

dotnet test --filter type=unit

this protects our builds from accidentally running integration tests that devs forgot to skip e.g. [Fact(Skip="Integration")], however it does require unit tests to "opt in" to CI by adding the correct traits which admittedly isn't great.

Upvotes: 14

richardwhatever
richardwhatever

Reputation: 5143

XUnit v2.0 is now available. Skippable tests are supported by it directly. Use:

[Fact (Skip = "specific reason")]

Upvotes: 332

Igor Zevaka
Igor Zevaka

Reputation: 76520

I would avoid externalising skipping tests (i.e. a config/command file if it's possible). This somewhat goes against making the tests easy to run and trustworthy. Making the tests ignored in code is the safest approach when other people start to get involved.

I could see a number of options, here are two that involve modification of existing code.

Option 1 - Most intrusive, compile time platform detection

In the VS Solution, define another configuration that defines a precompiler flag MONOWIN (just so that it's explicitly a flag the says that it is for code compiled on Windows for use on Mono).

Then define an attribute that will make the test ignored when compiled for Mono:

public class IgnoreOnMonoFactAttribute : FactAttribute {
#if MONOWIN
    public IgnoreOnMonoFactAttribute() {
        Skip = "Ignored on Mono";
    }
#endif
}

It's actually hard to find any advantages to this method as it involves mocking with the original solution and adds another confiration that needs to be supported.

Option 2 - somewhat intrusive - runtime platform detection

Here is a similar solution to option1, except no separate configuration is required:

public class IgnoreOnMonoFactAttribute : FactAttribute {

    public IgnoreOnMonoFactAttribute() {
        if(IsRunningOnMono()) {
            Skip = "Ignored on Mono";
        }
    }
    /// <summary>
    /// Determine if runtime is Mono.
    /// Taken from http://stackoverflow.com/questions/721161
    /// </summary>
    /// <returns>True if being executed in Mono, false otherwise.</returns>
    public static bool IsRunningOnMono() {
        return Type.GetType("Mono.Runtime") != null;
    }
}

Note 1

xUnit runner will run a method twice if it is marked with [Fact] and [IgnoreOnMonoFact]. (CodeRush doesn't do that, in this case I assume xUnit is correct). This means that any tests methods must have [Fact] replaced with [IgnoreOnMonoFact]

Note 2

CodeRush test runner still ran the [IgnoreOnMonoFact] test, but it did ignore the [Fact(Skip="reason")] test. I assume it is due to CodeRush reflecting xUnit and not actually running it with the aid of xUnit libraries. This works fine with xUnit runner.

Upvotes: 73

hoserdude
hoserdude

Reputation: 829

This is now solved in 1.8 - you can filter on Traits. See this issue log.

Update: Traits work with the console runner but not MSBuild, I've added a feature request for this support.

Upvotes: -2

Related Questions