How do you test an asynchronous method?

Question

I have an object that fetches XML or JSON over a network. Once this fetching is complete it calls a selector, passing in the returned data. So, for example I'd have something like:

-(void)testResponseWas200
{
    [MyObject get:@"foo.xml" withTarget:self selector:@selector(dataFinishedLoading:)];  
}

I tried the route of implementing dataFinishedLoading in the Test class and attempting to test inside that method, but the test suite is just locking up. This seems like it's a case for mocking, but I'm wondering if others have encountered this and how they handled it.

FYI: I'm using gh-unit for testing and any method prefixed with test* is executed automatically.

Answer 1

@jano Thank you I made of this little util from your post

In PYTestsUtils.m

+ (void)waitForBOOL:(BOOL*)finished forSeconds:(int)seconds {
    NSDate *timeout = [NSDate dateWithTimeIntervalSinceNow:seconds];
    while (!*finished && [timeout timeIntervalSinceNow]>0) {
        [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                                 beforeDate:[NSDate dateWithTimeIntervalSinceNow:1]];
    }
}

in my test file

- (void)testSynchronizeTime
{
    __block BOOL finished = NO;
    [self.connection synchronizeTimeWithSuccessHandler:^(NSTimeInterval serverTime) {
        NSLog(@"ServerTime %f", serverTime);
        finished = YES;
    } errorHandler:^(NSError *error) {
        STFail(@"Cannot get ServerTime %@", error);
        finished = YES;
    }];

    [PYTestsUtils waitForBOOL:&finished forSeconds:10];
    if (! finished)
        STFail(@"Cannot get ServerTime within 10 seconds");

}

variation

add in PYTestsUtils.m

+ (void)execute:(PYTestExecutionBlock)block ifNotTrue:(BOOL*)finished afterSeconds:(int)seconds {
    [self waitForBOOL:finished forSeconds:seconds];
    if (! *finished) block();
}

usage:

- (void)testSynchronizeTime
{
    __block BOOL finished = NO;
    [self.connection synchronizeTimeWithSuccessHandler:^(NSTimeInterval serverTime) {
        NSLog(@"ServerTime %f", serverTime);
        finished = YES;
    } errorHandler:^(NSError *error) {
        STFail(@"Cannot get ServerTime %@", error);
        finished = YES;
    }];

    [PYTestsUtils execute:^{
        STFail(@"Cannot get ServerTime within 10 seconds");
    } ifNotTrue:&finished afterSeconds:10];

}

Answer 2

Three ways that come to mind are: NSRunLoop, semaphores, and groups.

NSRunLoop

__block bool finished = false;

// For testing purposes we create this asynchronous task 
// that starts after 3 seconds and takes 1 second to execute.
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0UL);
dispatch_time_t threeSeconds = dispatch_time(DISPATCH_TIME_NOW, 3LL * NSEC_PER_SEC);
dispatch_after(threeSeconds, queue, ^{ 
    sleep(1); // replace this with your task
    finished = true; 
});

// loop until the flag is set from inside the task
while (!finished) {
    // spend 1 second processing events on each loop
    NSDate *oneSecond = [NSDate dateWithTimeIntervalSinceNow:1];
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:oneSecond];
}

A NSRunLoop is a loop that processes events like network ports, keyboard, or any other input source you plug in, and returns after processing those events, or after a time limit. When there are no events to process, the run loop puts the thread to sleep. All Cocoa and Core Foundation applications have a run loop underneath. You can read more about run loops in Apple's Threading Programming Guide: Run Loops, or in Mike Ash Friday Q&A 2010-01-01: NSRunLoop Internals.

In this test, I'm just using the NSRunLoop to sleep the thread for a second. Without it, the constant looping in the while would consume 100% of a CPU core.

If the block and the boolean flag are created in the same lexical scope (eg: both inside a method), then the flag needs the __block storage qualifier to be mutable. Had the flag been a global variable, it wouldn't need it.

If the test crashes before setting the flag, the thread is stuck waiting forever. Add a time limit to avoid that:

NSDate *timeout = [NSDate dateWithTimeIntervalSinceNow:2];
while (!finished && [timeout timeIntervalSinceNow]>0) {
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode 
                             beforeDate:[NSDate dateWithTimeIntervalSinceNow:1]];
}
if (!finished) NSLog(@"test failed with timeout");

If you are using this code for unit testing, an alternative way to insert a timeout is to dispatch a block with an assert:

// taken from https://github.com/JaviSoto/JSBarrierOperationQueue/blob/master/JSBarrierOperationQueueTests/JSBarrierOperationQueueTests.m#L118
dispatch_time_t timeout = dispatch_time(DISPATCH_TIME_NOW, 2LL * NSEC_PER_SEC);
dispatch_after(timeout, dispatch_get_main_queue(), ^(void){
    STAssertTrue(done, @"Should have finished by now");
});

Semaphore

Similar idea but sleeping until a semaphore changes, or until a time limit:

dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);

// signal the semaphore after 3 seconds using a global queue
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0UL);
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 3LL*NSEC_PER_SEC), queue, ^{ 
    sleep(1);
    dispatch_semaphore_signal(semaphore);
});

// wait with a time limit of 5 seconds
dispatch_time_t timeout = dispatch_time(DISPATCH_TIME_NOW, 5LL*NSEC_PER_SEC);
if (dispatch_semaphore_wait(semaphore, timeout)==0) {
    NSLog(@"success, semaphore signaled in time");
} else {
    NSLog(@"failure, semaphore didn't signal in time");
}

dispatch_release(semaphore);

If instead we waited forever with dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER); we would be stuck until getting a signal from the task, which keeps running on the background queue.

Group

Now imagine you have to wait for several blocks. You can use an int as flag, or create a semaphore that starts with a higher number, or you can group the blocks and wait until the group is finished. In this example I do the later with just one block:

dispatch_group_t group = dispatch_group_create();
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0UL);

// dispatch work to the given group and queue
dispatch_group_async(group,queue,^{
    sleep(1); // replace this with your task
});

// wait two seconds for the group to finish
dispatch_time_t timeout = dispatch_time(DISPATCH_TIME_NOW, 2LL*NSEC_PER_SEC);
if (dispatch_group_wait(group, timeout)==0) {
    NSLog(@"success, dispatch group completed in time");
} else {
    NSLog(@"failure, dispatch group did not complete in time");
}

dispatch_release(group);

If for some reason (to clean up resources?) you want to run a block after the group is finished, use dispatch_group_notify(group,queue, ^{/*...*/});

Answer 3

Asynchronous callbacks often require a message loop to run. It is a frequent pattern to stop the message loop after callback was called in the test code. Otherwise the loop is just waiting for next tasks, and there will be none.

Answer 4

One of the best ways to test asynchronous and multi-threaded code is with event logging. Your code should log events at interesting or useful times. Often an event alone is enough information to prove that logic is working correctly. Somtimes events will need payloads, or other meta information so they can be paired or chained.

This is most useful when the run-time or the operating system supports an efficient and robust eventing mechanism. This enables your product to ship with events in the 'retail' version. In this scenario, your events are only enabled when you need to debug a problem, or run a unit test to prove thins are working correctly.

Having the events in the retail (production) code lets you test and debug on any platform. This is huge benefit over debug or 'checked' code.

Note, like asserts, be careful where you put events - they can be expensive if logged to often. But the good news is that modern OSes and some application frameworks support eventing mechanisms that support 10's of thousands of events easily. Some support taking a stack trace on selected events. This can be very powerful, but usually requires that symbols are available at some point in time - either at logging, or trace post processing time on the target system.