Is it possible to use XCTest unit tests without Xcode?

Question

I'd like to implement XCTest unit-tests, but don't really want to use XCode to do it. It's not obvious how to do this, and I'm wondering if this is even possible?

From what I've found so far, one could get the impression that XCTest is completely dependent on XCode. I've found the xcodebuild test command-line support, but that depends on finding an XCode project or workspace.

Have I any options here, or do I just rip out the existing SenTestingKit code and revert to some home-brew unit test code? I have some such code to hand, but it's not the Right Thing To Do.

---

Rationale/history:

This is not just me being old-skool. I have an Objective-C program which I last touched two years ago, for which I had developed a reasonable set of unit tests based on SenTestingKit. Now I come back to this code – I may at least have to rebuild the thing, because of intervening library changes – I discover that SenTestingKit has disappeared, to be replaced by XCTest. Oh well....

This code was not developed using XCode, so there isn't a .project file associated with it, and the tests were up to now happily managed using SenTestingKit's main programs, and a Makefile check target (that's partly being old-skool, again, partly a lack of fondness for IDEs, and partly this having been an experiment with Objective-C, so originally sticking with what I know).

Answer 1

This one works ok:

1. Build your .xctest target as usual. By default it will be added to Plugins of the host app, but the location is irrelevant.

2. Create a runner command line tool with the code below. Update: Xcode ships runner that is fairly standalone in /Applications/Xcode.app/Contents/Developer/usr/bin/xctest You may use this one, if you don't want to create your own simple runner.

3. Run the tool with the full path to your test suite.

Sample runner code:

#import <Foundation/Foundation.h>
#import <XCTest/XCTest.h>

int main(int argc, const char* argv[]) {
  @autoreleasepool {
    XCTestSuite *suite = [XCTestSuite testSuiteForBundlePath:
      [NSString stringWithUTF8String:argv[1]]];
    [suite runTest];

    // Note that XCTestSuite is very shy in terms of errors,
    // so make sure that it loaded anything indeed:
    if (!suite.testRun.testCaseCount) return 1;

    return suite.testRun.hasSucceeded;
  }
}

Answer 2

Thanks, @stanislaw-pankevich, for a great answer. Here, for completeness, I'm including (more-or-less) the complete test program which I ended up with, which includes a couple of extra details and comments.

(This is a complete program from my point of view, since it tests functions defined in util.h, which isn't included here)

File UtilTest.h:

#import <XCTest/XCTest.h>
@interface UtilTest : XCTestCase
@end

File UtilTest.m:

#import "UtilTest.h"
#import "../util.h"  // the definition of the functions being tested

@implementation UtilTest

// We could add methods setUp and tearDown here.
// Every no-arg method which starts test... is included as a test-case.

- (void)testPathCanonicalization
{
    XCTAssertEqualObjects(canonicalisePath("/p1/./p2///p3/..//f3"), @"/p1/p2/f3");
}
@end

Driver program runtests.m (this is the main program, which the makefile actually invokes to run all the tests):

#import "UtilTest.h"
#import <XCTest/XCTestObservationCenter.h>

// Define my Observation object -- I only have to do this in one place
@interface BrownieTestObservation : NSObject<XCTestObservation>
@property (assign, nonatomic) NSUInteger testsFailed;
@property (assign, nonatomic) NSUInteger testsCalled;
@end

@implementation BrownieTestObservation

- (instancetype)init {
    self = [super init];
    self.testsFailed = 0;
    return self;
}

// We can add various other functions here, to be informed about
// various events: see XCTestObservation at
// https://developer.apple.com/reference/xctest?language=objc
- (void)testSuiteWillStart:(XCTestSuite *)testSuite {
    NSLog(@"suite %@...", [testSuite name]);
    self.testsCalled = 0;
}

- (void)testSuiteDidFinish:(XCTestSuite *)testSuite {
    NSLog(@"...suite %@ (%tu tests)", [testSuite name], self.testsCalled);
}

- (void)testCaseWillStart:(XCTestSuite *)testCase {
    NSLog(@"  test case: %@", [testCase name]);
    self.testsCalled++;
}

- (void)testCase:(XCTestCase *)testCase didFailWithDescription:(NSString *)description inFile:(NSString *)filePath atLine:(NSUInteger)lineNumber {
    NSLog(@"  FAILED: %@, %@ (%@:%tu)", testCase, description, filePath, lineNumber);
    self.testsFailed++;
}

@end

int main(int argc, char** argv) {
    XCTestObservationCenter *center = [XCTestObservationCenter sharedTestObservationCenter];
    BrownieTestObservation *observer = [BrownieTestObservation new];
    [center addTestObserver:observer];

    Class classes[] = { [UtilTest class], };  // add other classes here
    int nclasses = sizeof(classes)/sizeof(classes[0]);

    for (int i=0; i<nclasses; i++) {
        XCTestSuite *suite = [XCTestSuite testSuiteForTestCaseClass:classes[i]];
        [suite runTest];
    }

    int rval = 0;
    if (observer.testsFailed > 0) {
        NSLog(@"runtests: %tu failures", observer.testsFailed);
        rval = 1;
    }

    return rval;
}

Makefile:

FRAMEWORKS=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks
TESTCASES=UtilTest

%.o: %.m
    clang -F$(FRAMEWORKS) -c $<

check: runtests
    ./runtests 2>runtests.stderr

runtests: runtests.o $(TESTCASES:=.o) ../libmylib.a
    cc -o $@ $< -framework Cocoa -F$(FRAMEWORKS) -rpath $(FRAMEWORKS) \
        -framework XCTest $(TESTCASES:=.o) -L.. -lmylib

Notes:

• The XCTestObserver class is now deprecated, and replaced by XCTestObservation.
• The results of tests are sent to a shared XCTestObservationCenter, which unfortunately chatters distractingly to stderr (which therefore has to be redirected elsewhere) – it doesn't seem possible to avoid that and have them sent only to my observation centre instead. In my actual program, I replaced the NSLog calls in runtests.m with a function which chatters to stdout, which I could therefore distinguish from the chatter going to the default ObservationCenter.
• See also the overview documentation (presumes that you're using XCode),
• ...the XCTest API documentation,
• ...and the notes in the headers of the files at (eg) /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks/XCTest.framework/Headers

Answer 3

If you are looking for Xcode-based solutions see this and its linked solutions for examples.

For complete non-Xcode-based solution continue reading.

I used to ask similar answer a few years ago: Is there any non-Xcode-based command line unit testing tool for Objective-C? but things changed since then.

One interesting feature that appeared in XCTest over time is ability to run your custom test suites. I used to implement them successfully for my research needs, here is an example code which is a command line Mac OS application:

@interface FooTest : XCTestCase
@end

@implementation FooTest
- (void)testFoo {
  XCTAssert(YES);
}
- (void)testFoo2 {
  XCTAssert(NO);
}

@end

@interface TestObserver : NSObject <XCTestObservation>
@property (assign, nonatomic) NSUInteger testsFailed;
@end

@implementation TestObserver

- (instancetype)init {
  self = [super init];

  self.testsFailed = 0;

  return self;
}

- (void)testBundleWillStart:(NSBundle *)testBundle {
  NSLog(@"testBundleWillStart: %@", testBundle);
}

- (void)testBundleDidFinish:(NSBundle *)testBundle {
  NSLog(@"testBundleDidFinish: %@", testBundle);
}

- (void)testSuiteWillStart:(XCTestSuite *)testSuite {
  NSLog(@"testSuiteWillStart: %@", testSuite);
}

- (void)testCaseWillStart:(XCTestCase *)testCase {
  NSLog(@"testCaseWillStart: %@", testCase);
}

- (void)testSuiteDidFinish:(XCTestSuite *)testSuite {
  NSLog(@"testSuiteDidFinish: %@", testSuite);
}

- (void)testSuite:(XCTestSuite *)testSuite didFailWithDescription:(NSString *)description inFile:(NSString *)filePath atLine:(NSUInteger)lineNumber {
  NSLog(@"testSuite:didFailWithDescription:inFile:atLine: %@ %@ %@ %tu",
        testSuite, description, filePath, lineNumber);
}

- (void)testCase:(XCTestCase *)testCase didFailWithDescription:(NSString *)description inFile:(NSString *)filePath atLine:(NSUInteger)lineNumber {
  NSLog(@"testCase:didFailWithDescription:inFile:atLine: %@ %@ %@ %tu",
        testCase, description, filePath, lineNumber);
  self.testsFailed++;
}

- (void)testCaseDidFinish:(XCTestCase *)testCase {
  NSLog(@"testCaseWillFinish: %@", testCase);
}

@end

int RunXCTests() {
  XCTestObserver *testObserver = [XCTestObserver new];

  XCTestObservationCenter *center = [XCTestObservationCenter sharedTestObservationCenter];
  [center addTestObserver:testObserver];

  XCTestSuite *suite = [XCTestSuite defaultTestSuite];

  [suite runTest];

  NSLog(@"RunXCTests: tests failed: %tu", testObserver.testsFailed);

  if (testObserver.testsFailed > 0) {
    return 1;
  }

  return 0;
}

To compile this kind of code you will need to show a path to the folder where XCTest is located something like:

# in your Makefile
clang -F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks XCTestDriver.m

Don't expect the code to compile but it should give you an idea. Feel free to ask if you have any questions. Also follow the headers of XCTest framework to learn more about its classes and their docs.