Individual unit tests for each output of a method

Question

I have a method that takes a file as input and then returns N outputs based on this file.

I want to test this method in the following way: Say we have M files to test. For each file, I want to add one line to the test program (or to a separate file), consisting of the file path and the N expected outputs. This data should give rise to N*M individual tests, one for each pair of file and expected output.

Is there a good way to achieve this? I want each file to be parsed no more than once for each test run.

Below is an example that does what I want. As you can see, I have to add individual test classes for each file. I hope to find a solution where I can add just the line with the test data (e.g. testData.Add(("thirdfile", 4), (348, 312));) to test a new file.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Collections.Generic;

namespace ConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
        }
    }

    public static class FileParser
    {
        private static int n = 0;

        public static void Init(int parameter)
        {
            n = parameter;
        }

        public static (int output1, int output2) ParseFile(string filename)
        {
            return (filename[0] * n, filename[1] * n);
        }
    }

    public class Tests
    {
        private Dictionary<(string, int), (int, int)> testData;

        public Tests()
        {
            testData = new Dictionary<(string, int), (int, int)>();
            testData.Add(("somefile", 3), (345, 333));
            testData.Add(("anotherfile", 4), (291, 330));
            testData.Add(("thirdfile", 4), (348, 312));
        }

        public void TestOutput1((int, int) result, string filename, int parameter)
        {
            Assert.AreEqual(testData[(filename, parameter)].Item1, result.Item1);
        }

        public void TestOutput2((int, int) result, string filename, int parameter)
        {
            Assert.AreEqual(testData[(filename, parameter)].Item2, result.Item2);
        }
    }

    [TestClass]
    public class Somefile
    {
        protected static (int, int) fileParseResult;

        [ClassInitialize]
        public static void ClassInit(TestContext context)
        {
            FileParser.Init(3);
            fileParseResult = FileParser.ParseFile("somefile");
        }

        [TestMethod]
        public void SomefileOutput1() { var tests = new Tests(); tests.TestOutput1(fileParseResult, "somefile", 3); }
        [TestMethod]
        public void SomefileOutput2() { var tests = new Tests(); tests.TestOutput2(fileParseResult, "somefile", 3); }
    }

    [TestClass]
    public class Anotherfile
    {
        protected static (int, int) fileParseResult;

        [ClassInitialize]
        public static void ClassInit(TestContext context)
        {
            FileParser.Init(3);
            fileParseResult = FileParser.ParseFile("anotherfile");
        }

        [TestMethod]
        public void AnotherfileOutput1() { var tests = new Tests(); tests.TestOutput1(fileParseResult, "anotherfile", 4); }
        [TestMethod]
        public void AnotherfileOutput2() { var tests = new Tests(); tests.TestOutput2(fileParseResult, "anotherfile", 4); }
    }

    [TestClass]
    public class Thirdfile
    {
        protected static (int, int) fileParseResult;

        [ClassInitialize]
        public static void ClassInit(TestContext context)
        {
            FileParser.Init(3);
            fileParseResult = FileParser.ParseFile("thirdfile");
        }

        [TestMethod]
        public void ThirdfileOutput1() { var tests = new Tests(); tests.TestOutput1(fileParseResult, "thirdfile", 4); }
        [TestMethod]
        public void ThirdfileOutput2() { var tests = new Tests(); tests.TestOutput2(fileParseResult, "thirdfile", 4); }
    }
}

Answer 1

In case you're open to more than MSTest or xUnit, you can checkout Nuclear.Test.

Create a data driven test method that takes care of parsing and checking both result items at once.

[TestMethod]
[TestParamters("someFile", 3, (345, 333))]
[TestParamters("anotherfile", 4, (291, 330))]
[TestParamters("thirdfile", 4, (348, 312))]
void TestFile(String someFile, Int32 parameter, (Int32, Int32) expected) {

    (Int32, Int32) result = null;
    
    Test.Note("Parsing '" + someFile + "'");
    Test.IfNot.Action.ThrowsException(() => FileParser.Init(parameter), out Exception ex);
    Test.IfNot.Action.ThrowsException(() => result = FileParser.ParseFile(someFile), out ex);
    Test.IfNot.Object.IsNull(result);
    
    Test.Note("Checking results for '" + someFile + "'");
    Test.If.Value.IsEqual(result.Item1, expected.Item1);
    Test.If.Value.IsEqual(result.Item2, expected.Item2);
    
}

More information on writing data driven tests with Nuclear.Test can be found here.

Please note that this requires at least .NETStandard 2.0 at the moment. I do realize that you may not be open to a different unit testing platform, however since you did say you're open to either MSTest or xUnit i guess you haven't quite decided yet.

This approach also works with MSTest and xUnit, however this would break OAPT in these scenarios whereas Nuclear.Test is immune to those restrictions.

Please note that i wasn't able to test this code since i'm writing from my phone. There may be typos or a bug in there.

UPDATE:

this will be a legit approach using MSTest:

[TestMethod]
[DataRow("someFile", 3, (345, 333))]
[DataRow("anotherfile", 4, (291, 330))]
[DataRow("thirdfile", 4, (348, 312))]
public void TestFileParser(String fileName, Int32 parameter, (Int32, Int32) expected) {

    FileParser.Init(parameter);
    var result = FileParser.ParseFile(fileName);
    Assert.AreEqual(result, expected);

}

Turns out ValueTuple implements IComparable<(T1, T2)> and can be compared in one assert.

Answer 2

Using xUnit and System Linq

Using InlineData (or MemberData if you want it separate) containing both of your test results covers your requirement for adding one line of data to run multiple checks, however I'm not sure that

Each file should only be parsed once, not once per test.

is possible without some way of recording what files you've parsed in previous test runs, and plugging that in to the if() statement

public class ExampleTest
{
    [Theory]
    [InlineData ("somefile", 3, 332, 354)]
    [InlineData ("anotherfile", 3, 290, 337)]
    [InlineData ("thirdfile", 4, 310, 304)]
    public void FileParseOutputIsCorrect ( string fileName, int parameter, int resultA, int resultB )
    {
        //conditional check only necessary if you want to stop parsing in future test runs
        if ( !fileName.Parsed )
        {
            var fileParseResult = FileParser.ParseFile ( fileName, parameter );
            Assert.Equal ( fileParseResult[0], resultA );
            Assert.Equal ( fileParseResult[1], resultB );
        }
        else
        {
            Console.WriteLine ( $"Already parsed {fileName}" );
        }
    }
}

Answer 3

I agree with @Andreas to keep a unit test simple. Therefore reading a configuration file (a file that configures what the unit test does) would not be advisable. The following sample code extends @James Pusateri good answer by ensuring that evey file is read just once.

    [TestClass]
    public class UnitTest1
    {
        // Use a static Lazy<T> instance to read your file just once. 
        // Replace <object> with your type. 
        // Use multiple of these Lazy variables by using a Dictionary<string, Lazy<YourResultType>>
        private static Lazy<object> fileParseResult = new Lazy<object>(() => FileParser.ParseFile("somefile"));

        [ClassCleanup]
        public static void ClassCleanup()
        {
            // in case you need to clean up something, do it here
            // fileParseResult.Value.Dispose() if applicable
            fileParseResult = null;
        }

        [TestMethod]
        [DataRow("somefile", 3, 345, 333)]
        [DataRow("anotherfile", 4, 291, 330)]
        // add additional DataRow-lines here as required
        public void OutputIsValid(string fileName, int parameter, int resultX, int resultY)
        {
            // make sure to only read 'fileParseResult.Value' and not change it.
            Assert.AreEqual(fileParseResult.Value, fileName);
        }

        // dummy implementation for testing this code. Use your implemenation instead.
        private class FileParser
        {
            internal static object ParseFile(string v) => v;
        }
    }

MSTest This is a nice cheat-sheet https://www.automatetheplanet.com/mstest-cheat-sheet/

xUnit has a similar approach but uses different attributes [Theory] and [InlineData]. Also xUnit has more sophisticated (and complex) possibilities of context sharing https://xunit.net/docs/shared-context. Up to now, I've always managed to simplify the test scenarios in a way that these advanced context sharing features were not required.

Experience and personal opinion Actually, I do not use any context sharing for unit tests. The reason is refactoring. Consider the need to add another test method which requires a slightly different shared context. So you'd go ahead and change the shared context and unintendenly break a number of existing tests. Also I avoid reading any files inside unit tests, but rather use mocks that return a predefined and predictable result.

Answer 4

Well there is a way to make this possible with less code, following @pwrigshihanomoronimos approach. But then you will most likely need unit tests to make sure, the tests are working properly, so I would advice against it.

I know, writing such tests can be very tedious, but with unit tests there is one rule above all.

Keep them as simple as possibly possible.

Just use the bare minimum of complexity, which is absolutely necessary to make the test possible at all.

The less complex, the less error prone.

Better to have a separate file, with all the parameters and outputs to compare against and to read it up front.

Answer 5

Since your Output1 and Output2 methods are quite similar, you can use inheritance approach. Then you can apply test parameter insertion, if required

public class BaseTest
{
    private readonly string fileName;

    public BaseTest(string fileName)
    {
        this.fileName = fileName;
    }

    [ClassInitialize]
    public void Initialize()
    {
        // do your work with fileName
    }

    [TestCase]
    public void TestOutput1()
    {
        // test body
    }

    [TestCase]
    public void TestOutput2()
    {
        // test body
    }
}

[TestClass]
public class TestFile1 : BaseTest
{
    public TestFile1() : base("file1")
    {
    }
}

[TestClass]
public class TestFile2 : BaseTest
{
    public TestFile2() : base("file2")
    {
    }
}

Answer 6

You may actually be able to simplify this so that new tests on that library wouldn't necessarily require a code change to the test library itself.

MS documentation for data driven unit tests can be found here.

I have seen people use something like that with a csv file, then when a new test is needed, they simply add a row to the csv file.

Alternatively, I'd personally like the DataRow feature available in MSTest. Example MS Doc can be found here. I prefer this option, though a new test case does require a new line of code.

It should reduce the amount of code overall. Something a bit like this.

[TestClass]
public class FileClass
{
    [TestMethod]
    [DataRow("somefile", 3, 345, 333)]
    [DataRow("anotherfile", 4, 291, 330)]
    public void Output1IsValid(string fileName, int parameter, int resultX, int resultY) 
    { 
        var fileParseResult = FileParser.ParseFile(fileName);
        Assert.AreEqual(fileParseResult.Item1, resultX);         
    }

}