CODEWITHSUNDEEP
c#unit-testingcontinuous-integrationxunitcakebuild
Venkat
Venkat

Reputation: 2579

How to get passed and fail test case count in xunit using cake(c# make) script

I try to use the cake script for running the test cases written in Xunit using the cake script , I need to know the number of passed and failed test cases count.

#tool "nuget:?package=xunit.runner.console"
var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies);

Reference : http://www.cakebuild.net/dsl/xunit-v2

Can anyone please suggest how to get the number of passed and failed test cases?

Upvotes: 6

Views: 3355

Answers (2)

Gary Ewan Park
Gary Ewan Park

Reputation: 19011

Like most test runners, XUnit returns the number of failed tests in the return code from the console runner. Out of the box, Cake throws an exception, and therefore fails the build, when the return code of a tool is NOT zero.

This can be seen in the XUnit Runner Tests here:

https://github.com/cake-build/cake/blob/08907d1a5d97b66f58c01ae82506280882dcfacc/src/Cake.Common.Tests/Unit/Tools/XUnit/XUnitRunnerTests.cs#L145

Therefore, in order to know whether:

simply it is passed or failed in code level

This is known implicitly by whether or not the build succeeded or not. I typically use a strategy similar to this:

Task("Tests")
.Does(() =>
{
    var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
    XUnit2(testAssemblies,
        new XUnit2Settings {
            Parallelism = ParallelismOption.All,
            HtmlReport = false,
            NoAppDomain = true,
            XmlReport = true,
            OutputDirectory = "./build"
    });
})
.ReportError(exception =>
{
    Information("Some Unit Tests failed...");
    ReportUnit("./build/report-err.xml", "./build/report-err.html");
});

This is making use of the Exception Handling capabilities in Cake:

http://cakebuild.net/docs/fundamentals/error-handling

To take action when an error occurs. On top of that, I am then using the ReportUnit alias to convert the XML Report into a human readable HTML Report.

Upvotes: 1

Old Fox
Old Fox

Reputation: 8725

You'll have to use XUnit2Aliases​.XUnit2(IEnumerable < FilePath >, ​XUnit2Settings) + XmlPeekAliases for reading the XUnit output.

var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies,
     new XUnit2Settings {
        Parallelism = ParallelismOption.All,
        HtmlReport = false,
        NoAppDomain = true,
        XmlReport = true,
        OutputDirectory = "./build"
    });

The xml format is:(XUnit documentation, the example source, more information in Reflex)

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="1" errors="1" failures="0" skip="0">
    <testcase classname="path_to_test_suite.TestSomething"
              name="test_it" time="0">
        <error type="exceptions.TypeError" message="oops, wrong type">
        Traceback (most recent call last):
        ...
        TypeError: oops, wrong type
        </error>
    </testcase>
</testsuite>

Then the following snippet should bring you the information:

var file = File("./build/report-err.xml");
var failuresCount = XmlPeek(file, "/testsuite/@failures");
var testsCount = XmlPeek(file, "/testsuite/@tests");
var errorsCount = XmlPeek(file, "/testsuite/@errors");
var skipCount = XmlPeek(file, "/testsuite/@skip");

Upvotes: 11

Related Questions