Parametrized unit tests in Swift

Question

Is there any way to use parameterized unit tests, similar to what you can achieve in .Net using NUnit framework.

[TestCase(12, 3, 4)]
[TestCase(12, 2, 6)]
[TestCase(12, 4, 3)]
public void DivideTest(int expectedResult, int a, int b)
{
  Assert.AreEqual(expectedResult, a / b);
}

Using this kind of tests (vs non-parameterized ones) can give you bigger back for buck by allowing you to avoid writing series of almost identical unit tests differing only by parameter values.

I am looking for either XCTest-based solution or some other means to achieve it. Optimal solution should report each test case (parameter set) as a separate unit test in Xcode, so is it clear whether all or only some of the tests cases failed.

Answer 1

This is now available as part of the "Swift Testing" framework. Example from the docs:

@Test("Can make large orders", arguments: zip(Food.allCases, 1 ... 100))
func makeLargeOrder(of food: Food, count: Int) async throws {
  let foodTruck = FoodTruck(selling: food)
  #expect(await foodTruck.cook(food, quantity: count))
}

Source: https://developer.apple.com/documentation/testing/parameterizedtesting

Answer 2

With the advent of Swift Macros, you can do exactly what you want in pure Swift. My colleague and I have developed a neat Swift Macro for parameterized XCTest. With this, you can simply use:

@Parametrize(input: [(3, 4), (2, 6), (4, 3)], output: [12, 12, 12])
func testDivide(input numbers: (Int, Int), output result: Int) {
  XCTAssertEqual(numbers.0 / numbers.1, result)
}

The macro will create a copy of your method for a given parameter set. Check it out here: https://github.com/PGSSoft/XCTestParametrizedMacro

Answer 3

The best way to handle parameterized testing is to use XCTContext.runActivity. This allows to create a new activity with some name that will help you identify exactly which of the iterations failed. So for your division scenario:

func testDivision() {
    let testCases = [
        (a: 12, b: 3, result: 4),
        (a: 12, b: 2, result: 6),
        (a: 12, b: 6, result: 1),
        (a: 12, b: 4, result: 3),
    ]
    for (a, b, result) in testCases {
        XCTContext.runActivity(named: "Testing dividing \(a) by \(b) to get result \(result)") { activity in
            XCTAssertEqual(result, a/b)
        }
    }
}

Note that after running the above test, case no. 1, 2 and 4 will succeed while case no. 3 will fail. You can view exactly which test activity failed and which of the assertions caused faiure in test report:

Answer 4

The best way to use parametrized is using the XCTestCase subclass's property defaultTestSuite. A clear example with division is the following:

import XCTest

class ParameterizedExampleTests: XCTestCase {
    
    //properties to save the test cases
    private var array: [Float]? = nil
    private var expectedResult: Float? = nil
    
    // This makes the magic: defaultTestSuite has the set of all the test methods in the current runtime
    // so here we will create objects of ParameterizedExampleTests to call all the class' tests methodos
    // with differents values to test
    override open class var defaultTestSuite: XCTestSuite {
        let testSuite = XCTestSuite(name: NSStringFromClass(self))
        addTestsWithArray([12, 3], expectedResult: 4, toTestSuite: testSuite)
        addTestsWithArray([12, 2], expectedResult: 6, toTestSuite: testSuite)
        addTestsWithArray([12, 4], expectedResult: 3, toTestSuite: testSuite)
        return testSuite
    }
    
    // This is just to create the new ParameterizedExampleTests instance to add it into testSuite
    private class func addTestsWithArray(_ array: [Float], expectedResult: Float, toTestSuite testSuite: XCTestSuite) {
        testInvocations.forEach { invocation in
            let testCase = ParameterizedExampleTests(invocation: invocation)
            testCase.array = array
            testCase.expectedResult = expectedResult
            testSuite.addTest(testCase)
        }
    }

    // Normally this function is into production code (e.g. class, struct, etc).
    func division(a: Float, b: Float) -> Float {
        return a/b
    }
    
    func testDivision() {
        XCTAssertEqual(self.expectedResult, division(a: array?[0] ?? 0, b: array?[1] ?? 0))
    }
}

Answer 5

We found that this solution How to Dynamically add XCTestCase offers us the flexibility we need. Being able to dynamically add tests, pass arguments to tests, and display the dynamic test names in the test report.

Another option is to checkout XCTestPlan in XCode. There's an informative video from WWDC about it.

Answer 6

I rather like @DariusV's solution. However, it doesn't handle well when I the developer execute the test method directly from Xcode's sidebar. That's a dealbreaker for me.

What I wound up doing I think is rather slick.

I declare a Dictionary of testValues (probs need a better name for that) as an instance computed property of my XCTestCase subclass. Then, I define a Dictionary literal of inputs keying expected outputs. My example tests a function that acts on Int, so I define testValues like so:

static var testRange: ClosedRange<Int> { 0...100 }

var testValues: [Int: Int] {
    let range = Self.testRange
    return [
        // Lower extreme
        Int.min: range.lowerBound,

        // Lower bound
        range.lowerBound - 1: range.lowerBound,
        range.lowerBound    : range.lowerBound,
        range.lowerBound + 1: range.lowerBound + 1,

        // Middle
        25: 25,
        50: 50,
        75: 75,

        // Upper bound
        range.upperBound - 1: range.upperBound - 1,
        range.upperBound    : range.upperBound,
        range.upperBound + 1: range.upperBound,

        // Upper extreme
        Int.max: range.upperBound
    ]
}

Here, I very easily declare my edge and boundary cases. A more semantic way of accomplishing the same might be to use an array of tuples, but Swift's dictionary literal syntax is thin enough, and I know what this does. 😊

Now, in my test method, I have a simple for loop.

/// The property wrapper I'm testing. This could be anything, but this works for example.
@Clamped(testRange) var number = 50

func testClamping() {
    let initial = number

    for (input, expected) in testValues {
        // Call the code I'm testing. (Computed properties act on assignment)
        number = input
        XCTAssertEqual(number, expected, "{number = \(input)}: `number` should be \(expected)")

        // Reset after each iteration.
        number = initial
    }
}

Now to run for each parameter, I simply invoke XCTests in any of Xcode's normal ways, or any other way that works with Linux (I assume). No need to run every test class to get this one's parameterizedness.

Isn't that pretty? I just covered every boundary value and equivalence class in only a few lines of DRY code!

As for identifying failing cases, each invocation runs through an XCTAssert function, which as per Xcode's convention will only throw a message at you if there's something wrong that you need to think about. These show up in the sidebar, but similar messages tend to blend together. My message string here identifies the failing input and its resulting output, fixing the blending-togetherness, and making my testing flow a sane piece of cherry apple pie. (You may format yours any way you like, bumpkin! Whatever blesses your sanity.)

Delicious.

TL;DR

An adaptation of @Code Different's answer: Use a dictionary of inputs and outputs, and run with a for loop. 😄

Answer 7

The asserts all seem to throw, so perhaps something like this will work for you:

typealias Division = (dividend: Int, divisor: Int, quotient: Int)

func testDivisions() {
    XCTAssertNoThrow(try divisionTest((12, 3, 4)))
    XCTAssertNoThrow(try divisionTest((12, 2, 6)))
    XCTAssertNoThrow(try divisionTest((12, 4, 3)))
}

private func divisionTest(_ division: Division) throws {
    XCTAssertEqual(division.dividend / division.divisor, division.quotient)
}

If one fails, the entire function will fail. If more granularity is required, every case can be split up into an individual function.

Answer 8

You function parameters are all over the place. I'm not sure if your function is doing multiplication or division. But here's one way you can do multiple test cases in a single test method.

Given this function:

func multiply(_ a: Int, _ b: Int) -> Int {
    return a * b
}

You can have multiple test cases on it:

class MyTest: XCTestCase {
    func testMultiply() {
        let cases = [(4,3,12), (2,4,8), (3,5,10), (4,6,20)]
        cases.forEach {
            XCTAssertEqual(multiply($0, $1), $2)
        }
    }
}

The last two would fail and Xcode will tell you about them.

Answer 9

@Code Different's answer is legit. Here's other two options, or rather workarounds:

Property Based Testing

You could use a tool like Fox to perform generative testing, where the test framework will generate many input sets for the behaviour you want to test and run them for you.

More on this approach:

• http://www.pivotaltracker.com/community/tracker-blog/generative-testing
• http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html

BDD Shared Examples

If you like the BDD style and are using a test framework that supports them, you could use shared examples.

Using Quick it would look like:

class MultiplySharedExamplesConfiguration: QuickConfiguration {
  override class func configure(configuration: Configuration) {
    sharedExamples("something edible") { (sharedExampleContext: SharedExampleContext) in
      it("multiplies two numbers") {
        let a = sharedExampleContext()["a"]
        let b = sharedExampleContext()["b"]
        let expectedResult = sharedExampleContext()["b"]

        expect(multiply(a, b)) == expectedResult
      }
    }
  }
}

class MultiplicationSpec: QuickSpec {
  override func spec() {
    itBehavesLike("it multiplies two numbers") { ["a": 2, "b": 3, "result": 6] }
    itBehavesLike("it multiplies two numbers") { ["a": 2, "b": 4, "result": 8] }
    itBehavesLike("it multiplies two numbers") { ["a": 3, "b": 3, "result": 9] }
  }
}

To be honest this option is: 1) a lot of work, 2) a misuse of the shared example technique, as you are not using them to test behaviour shared by multiple classes but rather to parametrise the test. But as I said at the start, this is a more of a workaround.