Swift: Unit tests with functions that throw

Question

How can one write unit tests in Swift that communicate useful information when calling functions that can throw?

What I'd really like to be able to do is something like this:

class TestTestsTests: XCTestCase {

    func doFoo() throws -> String {
        // A complex operation that might throw in various places
        return "foo"
    }

    func doBar() throws -> String {
        // A complex operation that might throw in various places
        return "bar"
    }

    func testExample() throws {
        let foo = try doFoo()
        let bar = try doBar()
        XCTAssertNotEqual(foo, bar)
    }

}

Ideally the unit test runner would stop on the line where an unhandled exception occurred, and let the user explore the stack and error message. Unfortunately, adding throws to the test function causes it to be silently skipped over (and there's a UI bug that makes it look as though the test is still being run, and getting the same result as before adding throws).

Of course it is also possible to do this:

func testExample() {
    let foo = try! doFoo()
    let bar = try! doBar()
    XCTAssertNotEqual(foo, bar)
}

But now a failure doesn't really provide the context we need. Add a throw to doFoo, and we get a message like fatal error: 'try!' expression unexpectedly raised an error: TestTestsTests.Error(): file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-700.1.101.15/src/swift/stdlib/public/core/ErrorType.swift, line 50, which only gives us the line number within testExample, and not within doFoo where the error occurred. This also seems to get the debugger 'stuck' (clicking continue just returns us to the same line with the same error message) on that line, regardless of whether breakpoints are enabled, and prevents other tests from running.

So maybe we could try something like this?

func testExample() {
    do {
        let foo = try doFoo()
        let bar = try doBar()
        XCTAssertNotEqual(foo, bar)
    } catch {
        XCTFail("\(error)")
    }
}

This runs as expected, however we can't determine which of doFoo or doBar threw the error, and also no line number information. At least we can get the error message and not prevent other tests from running.

I could go on, but the short of it is that I can't find a way to simultaneously not break the unit test running (like with try!), figure out which function threw the error, and get the error information -- Unless I do something ridiculous like:

func testExample() {
    var foo: String? = nil
    var bar: String? = nil
    do {
        foo = try doFoo()
    } catch {
        XCTFail("\(error)")
        return
    }
    do {
        bar = try doBar()
    } catch {
        XCTFail("\(error)")
        return
    }

    if let foo = foo, bar = bar {
        XCTAssertNotEqual(foo, bar)
    }
}

And I still don't get to find out where in doFoo or doBar the error occurred.

Is this just the sad state of unit testing in Swift, or am I missing something?

Answer 1

This runs as expected, however we can't determine which of doFoo or doBar threw the error, and also no line number information.

Maybe this is just an old problem or I'm not understanding your problem, or maybe you're just kind of making a statement rather than asking a question. But with respect to the quoted statement above, you can add any information you like to the messages in XCTest Assertions. Swift also provides the following literals:

#file          String   Name of the file where this literal appears.
#line          Int      The line number on which it appears.
#column        Int      The column number in which it begins.
#function      String   The name of the declaration in which it appears. 


func testExample() {
    var foo: String? = nil
    var bar: String? = nil
    do {
        foo = try doFoo()
    } catch {
        XCTFail("try doFoo() failed on line: \(#line) in file: \(#file) with error: \(error)")
        return
    }
    do {
        bar = try doBar()
    } catch {
        XCTFail("try doBar() failed on line: \(#line) in file: \(#file) with error: \(error)")
        return
    }

    if let foo = foo, bar = bar {
        XCTAssertNotEqual(foo, bar)
    }
}

If you really want to go crazy with it you can add error handling to your doBar() method and that error can contain any internal information you'd like. In fact... by implementing your own errors in your methods you might not even need to separate the methods into two blocks in your tests, just printing the error should be enough. You can put any information you like in the error message.

Anyway, I think this is an outdated issue, you can get all the information you need from the test logs - they list out all the methods that failed and even have little arrows that let you jump right to the test that failed. They then highlight the specific assertion that failed... from there it's quite easy to tell what is happening in most cases. Worst case scenario you have to set a breakpoint or two and run the test again.

Answer 2

You can do your own errors, using Error or LocalizedError protocols

enum Errors: Error, CustomStringConvertible {
    case foo_param_is_null
    case bar_param_is_null(paramIndex: Int)

    var description: String {
        switch self {
        case .foo_param_is_null:
            return "Param is null in foo"
        case .bar_param_is_null(let paramIndex):
            return "Param at index \(paramIndex) is null in bar"
        }
    }
}

func foo(_ param: Int) throws {
    guard param != 0 else {
        throw Errors.foo_param_is_null
    }
    print("foo = \(param)")
}

func bar(_ params: [Int]) throws {
    if let index = params.firstIndex(where: {$0 == 0}) {
        throw Errors.bar_param_is_null(paramIndex: index)
    }
    print("bar = \(params)")
}

do {
    try foo(1)
    try foo(0)
} catch {
    print("\(error)")
}

do {
    try bar([1,2,3])
    try bar([1,0,3])
} catch {
    print("\(error)")
}

Result:

foo = 1
Param is null in foo
bar = [1, 2, 3]
Param at index 1 is null in bar

And if you need even more information, you can use structures to define errors and error domains. Something like :

struct FooBarError: Error, CustomStringConvertible {
    var string: String
    var context: Any?
    
    static func fooError() {
        FooBarError(string: "Foo Error")
    }
    
    static func barError(context: BarErrorContext) { FooBarError(string: "Bar Error", context: context)
    }
    
    var description: String {
        if let cox = context as? BarErrorContext {
            return "\(string) - paramIndex: \(ctx.paramIndex) - \(ctx.insidiousReason)"
        }
        return string
    }
}

Note:

As @ibrust proposed, you can pass #function, #line and other special parameters to your errors initialisers to provide this information

do {
    try foo()
} catch {
    throw(BarFooError.foo(line: #line))
}

You can also propagate the original error

do {
    try bar()
} catch {
    throw(BarFooError.bar(exception: error))
}

Edited:

Finally , you can also use print(Thread.callStackSymbols) in your error description, but at this point, there is a risk of confusion between debugging and testing. Just a personal thought.