Testing assertion in Swift

Question

I'm writing unit tests for a method that has an assertion. The Swift Language guide recommends using assertions for "invalid conditions":

Assertions cause your app to terminate and are not a substitute for designing your code in such a way that invalid conditions are unlikely to arise. Nonetheless, in situations where invalid conditions are possible, an assertion is an effective way to ensure that such conditions are highlighted and noticed during development, before your app is published.

I want to test the failure case.

However, there is not XCTAssertThrows in Swift (as of Beta 6). How can I write an unit test that tests that an assertion fails?

Edit

As per @RobNapier's suggestion, I tried wrapping XCTAssertThrows in an Objective-C method and calling this method from Swift. This doesn't work as the macro does not catch the fatal error caused by assert, and thus the test crashes.

Answer 1

We have Swift (4) code that tests an Objective-C framework. Some of the framework methods call into NSAssert.

Inspired by NSHipster, I ended up with an implementation like such:

SwiftAssertionHandler.h (use this in a bridging header)

@interface SwiftAssertionHandler : NSAssertionHandler

@property (nonatomic, copy, nullable) void (^handler)(void);

@end

SwiftAssertionHandler.m

@implementation SwiftAssertionHandler

- (instancetype)init {
    if (self = [super init]) {
        [[[NSThread currentThread] threadDictionary] setValue:self
                                                           forKey:NSAssertionHandlerKey];
    }
    return self;
}

- (void)dealloc {
    [[[NSThread currentThread] threadDictionary] removeObjectForKey:NSAssertionHandlerKey];
}

- (void)handleFailureInMethod:(SEL)selector object:(id)object file:(NSString *)fileName lineNumber:(NSInteger)line description:(NSString *)format, ... {
    if (self.handler) {
        self.handler();
    }
}

- (void)handleFailureInFunction:(NSString *)functionName file:(NSString *)fileName lineNumber:(NSInteger)line description:(NSString *)format, ... {
    if (self.handler) {
        self.handler();
    }
}

@end

Test.swift

let assertionHandler = SwiftAssertionHandler()
assertionHandler.handler = { () -> () in
    // i.e. count number of assert
}

Answer 2

Matt Gallagher's CwlPreconditionTesting project on github adds a catchBadInstruction function which gives you the ability to test for assertion/precondition failures in unit test code.

The CwlCatchBadInstructionTests file shows a simple illustration of its use. (Note that it only works in the simulator for iOS.)

Answer 3

Thanks to nschum and Ken Ko for the idea behind this answer.

Here is a gist for how to do it

Here is an example project

This answer is not just for assert. It's also for the other assertion methods (assert, assertionFailure, precondition, preconditionFailure and fatalError)

1. Drop ProgrammerAssertions.swift to the target of your app or framework under test. Just besides your source code.

ProgrammerAssertions.swift

import Foundation

/// drop-in replacements

public func assert(@autoclosure condition: () -> Bool, @autoclosure _ message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    Assertions.assertClosure(condition(), message(), file, line)
}

public func assertionFailure(@autoclosure message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    Assertions.assertionFailureClosure(message(), file, line)
}

public func precondition(@autoclosure condition: () -> Bool, @autoclosure _ message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    Assertions.preconditionClosure(condition(), message(), file, line)
}

@noreturn public func preconditionFailure(@autoclosure message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    Assertions.preconditionFailureClosure(message(), file, line)
    runForever()
}

@noreturn public func fatalError(@autoclosure message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    Assertions.fatalErrorClosure(message(), file, line)
    runForever()
}

/// Stores custom assertions closures, by default it points to Swift functions. But test target can override them.
public class Assertions {

    public static var assertClosure              = swiftAssertClosure
    public static var assertionFailureClosure    = swiftAssertionFailureClosure
    public static var preconditionClosure        = swiftPreconditionClosure
    public static var preconditionFailureClosure = swiftPreconditionFailureClosure
    public static var fatalErrorClosure          = swiftFatalErrorClosure

    public static let swiftAssertClosure              = { Swift.assert($0, $1, file: $2, line: $3) }
    public static let swiftAssertionFailureClosure    = { Swift.assertionFailure($0, file: $1, line: $2) }
    public static let swiftPreconditionClosure        = { Swift.precondition($0, $1, file: $2, line: $3) }
    public static let swiftPreconditionFailureClosure = { Swift.preconditionFailure($0, file: $1, line: $2) }
    public static let swiftFatalErrorClosure          = { Swift.fatalError($0, file: $1, line: $2) }
}

/// This is a `noreturn` function that runs forever and doesn't return.
/// Used by assertions with `@noreturn`.
@noreturn private func runForever() {
    repeat {
        NSRunLoop.currentRunLoop().run()
    } while (true)
}

2. Drop XCTestCase+ProgrammerAssertions.swift to your test target. Just besides your test cases.

XCTestCase+ProgrammerAssertions.swift

import Foundation
import XCTest
@testable import Assertions

private let noReturnFailureWaitTime = 0.1

public extension XCTestCase {

    /**
     Expects an `assert` to be called with a false condition.
     If `assert` not called or the assert's condition is true, the test case will fail.

     - parameter expectedMessage: The expected message to be asserted to the one passed to the `assert`. If nil, then ignored.
     - parameter file:            The file name that called the method.
     - parameter line:            The line number that called the method.
     - parameter testCase:        The test case to be executed that expected to fire the assertion method.
     */
    public func expectAssert(
        expectedMessage: String? = nil,
        file: StaticString = __FILE__,
        line: UInt = __LINE__,
        testCase: () -> Void
        ) {

            expectAssertionReturnFunction("assert", file: file, line: line, function: { (caller) -> () in

                Assertions.assertClosure = { condition, message, _, _ in
                    caller(condition, message)
                }

                }, expectedMessage: expectedMessage, testCase: testCase) { () -> () in
                    Assertions.assertClosure = Assertions.swiftAssertClosure
            }
    }

    /**
     Expects an `assertionFailure` to be called.
     If `assertionFailure` not called, the test case will fail.

     - parameter expectedMessage: The expected message to be asserted to the one passed to the `assertionFailure`. If nil, then ignored.
     - parameter file:            The file name that called the method.
     - parameter line:            The line number that called the method.
     - parameter testCase:        The test case to be executed that expected to fire the assertion method.
     */
    public func expectAssertionFailure(
        expectedMessage: String? = nil,
        file: StaticString = __FILE__,
        line: UInt = __LINE__,
        testCase: () -> Void
        ) {

            expectAssertionReturnFunction("assertionFailure", file: file, line: line, function: { (caller) -> () in

                Assertions.assertionFailureClosure = { message, _, _ in
                    caller(false, message)
                }

                }, expectedMessage: expectedMessage, testCase: testCase) { () -> () in
                    Assertions.assertionFailureClosure = Assertions.swiftAssertionFailureClosure
            }
    }

    /**
     Expects an `precondition` to be called with a false condition.
     If `precondition` not called or the precondition's condition is true, the test case will fail.

     - parameter expectedMessage: The expected message to be asserted to the one passed to the `precondition`. If nil, then ignored.
     - parameter file:            The file name that called the method.
     - parameter line:            The line number that called the method.
     - parameter testCase:        The test case to be executed that expected to fire the assertion method.
     */
    public func expectPrecondition(
        expectedMessage: String? = nil,
        file: StaticString = __FILE__,
        line: UInt = __LINE__,
        testCase: () -> Void
        ) {

            expectAssertionReturnFunction("precondition", file: file, line: line, function: { (caller) -> () in

                Assertions.preconditionClosure = { condition, message, _, _ in
                    caller(condition, message)
                }

                }, expectedMessage: expectedMessage, testCase: testCase) { () -> () in
                    Assertions.preconditionClosure = Assertions.swiftPreconditionClosure
            }
    }

    /**
     Expects an `preconditionFailure` to be called.
     If `preconditionFailure` not called, the test case will fail.

     - parameter expectedMessage: The expected message to be asserted to the one passed to the `preconditionFailure`. If nil, then ignored.
     - parameter file:            The file name that called the method.
     - parameter line:            The line number that called the method.
     - parameter testCase:        The test case to be executed that expected to fire the assertion method.
     */
    public func expectPreconditionFailure(
        expectedMessage: String? = nil,
        file: StaticString = __FILE__,
        line: UInt = __LINE__,
        testCase: () -> Void
        ) {

            expectAssertionNoReturnFunction("preconditionFailure", file: file, line: line, function: { (caller) -> () in

                Assertions.preconditionFailureClosure = { message, _, _ in
                    caller(message)
                }

                }, expectedMessage: expectedMessage, testCase: testCase) { () -> () in
                    Assertions.preconditionFailureClosure = Assertions.swiftPreconditionFailureClosure
            }
    }

    /**
     Expects an `fatalError` to be called.
     If `fatalError` not called, the test case will fail.

     - parameter expectedMessage: The expected message to be asserted to the one passed to the `fatalError`. If nil, then ignored.
     - parameter file:            The file name that called the method.
     - parameter line:            The line number that called the method.
     - parameter testCase:        The test case to be executed that expected to fire the assertion method.
     */
    public func expectFatalError(
        expectedMessage: String? = nil,
        file: StaticString = __FILE__,
        line: UInt = __LINE__,
        testCase: () -> Void) {

            expectAssertionNoReturnFunction("fatalError", file: file, line: line, function: { (caller) -> () in

                Assertions.fatalErrorClosure = { message, _, _ in
                    caller(message)
                }

                }, expectedMessage: expectedMessage, testCase: testCase) { () -> () in
                    Assertions.fatalErrorClosure = Assertions.swiftFatalErrorClosure
            }
    }

    // MARK:- Private Methods

    private func expectAssertionReturnFunction(
        functionName: String,
        file: StaticString,
        line: UInt,
        function: (caller: (Bool, String) -> Void) -> Void,
        expectedMessage: String? = nil,
        testCase: () -> Void,
        cleanUp: () -> ()
        ) {

            let expectation = expectationWithDescription(functionName + "-Expectation")
            var assertion: (condition: Bool, message: String)? = nil

            function { (condition, message) -> Void in
                assertion = (condition, message)
                expectation.fulfill()
            }

            // perform on the same thread since it will return
            testCase()

            waitForExpectationsWithTimeout(0) { _ in

                defer {
                    // clean up
                    cleanUp()
                }

                guard let assertion = assertion else {
                    XCTFail(functionName + " is expected to be called.", file: file.stringValue, line: line)
                    return
                }

                XCTAssertFalse(assertion.condition, functionName + " condition expected to be false", file: file.stringValue, line: line)

                if let expectedMessage = expectedMessage {
                    // assert only if not nil
                    XCTAssertEqual(assertion.message, expectedMessage, functionName + " called with incorrect message.", file: file.stringValue, line: line)
                }
            }
    }

    private func expectAssertionNoReturnFunction(
        functionName: String,
        file: StaticString,
        line: UInt,
        function: (caller: (String) -> Void) -> Void,
        expectedMessage: String? = nil,
        testCase: () -> Void,
        cleanUp: () -> ()
        ) {

            let expectation = expectationWithDescription(functionName + "-Expectation")
            var assertionMessage: String? = nil

            function { (message) -> Void in
                assertionMessage = message
                expectation.fulfill()
            }

            // act, perform on separate thead because a call to function runs forever
            dispatch_async(dispatch_get_global_queue(QOS_CLASS_USER_INITIATED, 0), testCase)

            waitForExpectationsWithTimeout(noReturnFailureWaitTime) { _ in

                defer {
                    // clean up
                    cleanUp()
                }

                guard let assertionMessage = assertionMessage else {
                    XCTFail(functionName + " is expected to be called.", file: file.stringValue, line: line)
                    return
                }

                if let expectedMessage = expectedMessage {
                    // assert only if not nil
                    XCTAssertEqual(assertionMessage, expectedMessage, functionName + " called with incorrect message.", file: file.stringValue, line: line)
                }
            }
    }
}

3. Use assert, assertionFailure, precondition, preconditionFailure and fatalError normally as you always do.

For example: If you have a function that does a division like the following:

func divideFatalError(x: Float, by y: Float) -> Float {

    guard y != 0 else {
        fatalError("Zero division")
    }

    return x / y
}

4. Unit test them with the new methods expectAssert, expectAssertionFailure, expectPrecondition, expectPreconditionFailure and expectFatalError.

You can test the 0 division with the following code.

func testFatalCorrectMessage() {
    expectFatalError("Zero division") {
        divideFatalError(1, by: 0)
    }
}

Or if you don't want to test the message, you simply do.

func testFatalErrorNoMessage() {
    expectFatalError() {
        divideFatalError(1, by: 0)
    }
}

Answer 4

Agree with nschum's comment that it doesn't seem right to unit test assert because by default it wont be in the prod code. But if you really wanted to do it, here is the assert version for reference:

override assert

func assert(@autoclosure condition: () -> Bool, @autoclosure _ message: () -> String = "", file: StaticString = __FILE__, line: UInt = __LINE__) {
    assertClosure(condition(), message(), file, line)
}
var assertClosure: (Bool, String, StaticString, UInt) -> () = defaultAssertClosure
let defaultAssertClosure = {Swift.assert($0, $1, file: $2, line: $3)}

helper extension

extension XCTestCase {

    func expectAssertFail(expectedMessage: String, testcase: () -> Void) {
        // arrange
        var wasCalled = false
        var assertionCondition: Bool? = nil
        var assertionMessage: String? = nil
        assertClosure = { condition, message, _, _ in
            assertionCondition = condition
            assertionMessage = message
            wasCalled = true
        }

        // act
        testcase()

        // assert
        XCTAssertTrue(wasCalled, "assert() was never called")
        XCTAssertFalse(assertionCondition!, "Expected false to be passed to the assert")
        XCTAssertEqual(assertionMessage, expectedMessage)

        // clean up
        assertClosure = defaultAssertClosure
    }
}

Answer 5

assert and its sibling precondition don't throw exceptions cannot be "caught" (even with Swift 2's error handling).

A trick you can use is to write your own drop-in replacement that does the same thing but can be replaced for tests. (If you're worried about performance, just #ifdef it away for release builds.)

custom precondition

/// Our custom drop-in replacement `precondition`.
///
/// This will call Swift's `precondition` by default (and terminate the program).
/// But it can be changed at runtime to be tested instead of terminating.
func precondition(@autoclosure condition: () -> Bool, @autoclosure _ message: () -> String = "", file: StaticString = __FILE__, line: UWord = __LINE__) {
    preconditionClosure(condition(), message(), file, line)
}

/// The actual function called by our custom `precondition`.
var preconditionClosure: (Bool, String, StaticString, UWord) -> () = defaultPreconditionClosure
let defaultPreconditionClosure = {Swift.precondition($0, $1, file: $2, line: $3)}

test helper

import XCTest

extension XCTestCase {
    func expectingPreconditionFailure(expectedMessage: String, @noescape block: () -> ()) {

        let expectation = expectationWithDescription("failing precondition")

        // Overwrite `precondition` with something that doesn't terminate but verifies it happened.
        preconditionClosure = {
            (condition, message, file, line) in
            if !condition {
                expectation.fulfill()
                XCTAssertEqual(message, expectedMessage, "precondition message didn't match", file: file.stringValue, line: line)
            }
        }

        // Call code.
        block();

        // Verify precondition "failed".
        waitForExpectationsWithTimeout(0.0, handler: nil)

        // Reset precondition.
        preconditionClosure = defaultPreconditionClosure
    }
}

example

func doSomething() {
    precondition(false, "just not true")
}

class TestCase: XCTestCase {
    func testExpectPreconditionFailure() {
        expectingPreconditionFailure("just not true") {
            doSomething();
        }
    }
}

(gist)

Similar code will work for assert, of course. However, since you're testing the behavior, you obviously want it to be part of your interface contract. You don't want optimized code to violate it, and assert will be optimized away. So better use precondition here.

Answer 6

I believe as of Beta6 it is still impossible for Swift to catch an exception directly. The only way you can handle this is to write that particular test case in ObjC.

That said, note that _XCTAssertionType.Throws does exist, which suggests that the Swift team is aware of this and intends eventually to provide a solution. It is quite imaginable that you could write this assertion yourself in ObjC and expose it to Swift (I can't think of any reason that would be impossible in Beta6). The one big problem is that you may not easily be able to get good location information out of it (the specific line that failed, for instance).