CODEWITHSUNDEEP
iosunit-testingxcode5xctest
Shinigami
Shinigami

Reputation: 2213

Xcode 5 says tests failed but shows green checkmarks

I have written a few unit tests in the new XCTest framework of Xcode 5.

The tests seem to pass successfully most of the time, but I've come across a strange behaviour where if I run the tests enough times consecutively, eventually I'll get a 'test failed' message and the console will show Program ended with exit code 0, but there are green checkmarks beside all the tests.

What's going wrong here?

Upvotes: 44

Views: 4241

Answers (9)

ReDetection
ReDetection

Reputation: 3196

Despite the test I'm using is logic tests, Xcode wants to iPhone simulator to be running. Once in the agony of trying to get tests working again I've reset the iPhone Simulator contents and Settings from it's menu, and this helped.

Upvotes: 0

HackaZach
HackaZach

Reputation: 409

This bug has been corrected in Xcode 5.1 :

Xcode 5.1 release notes

Notes specifically mention :

Test target [test name] encountered an error (Test process exited with code -1)

Attempt recovery by quitting and restarting the simulator. (15929053)

Upvotes: 0

Alex
Alex

Reputation: 1038

This appears to have been fixed in XCode 5.1.1

Upvotes: 0

rl09
rl09

Reputation: 81

Having just run into this in my own updated XCode 5 thought I'd share my similar resolutions.

I notice if I hit any breakpoint when running the test and simply continue it reports as passing, very odd indeed and unacceptable for running your whole suite...

I also noticed adding a usleep as someone mentioned, as well as doing a clean build prior to running the tests seems to resolve the issue. However creating a new test often re-introduces the issue at least for that new test. Hope Apple fixes this soon!

Upvotes: 0

RyanM
RyanM

Reputation: 4594

For me, I think this problem was linked to incorrect OCMock configuration. Removing OCMock (completely) and re-adding it to the project exactly as recommended by OCMock seems to have fixed it.

A couple specific things I did wrong:

  1. I was using #import "OCMock.h instead of #import "OCMock/OCMock.h. Xcode autocompleted the former.
  2. I had a bad link leftover in library search paths from previously trying to install OCMock with a different directory structure.
  3. This is dumb, but in trying to debug this issue, I commented out the instantiation of my OCMockObject in my setup method.. and didn't remember to uncomment it. The nil OCMockObject caused similar behavior as described in the other answers here. This is odd.. In my current project I have two tests that use the OCMockObject that is instantiated in my setup. Both should fail. However, one of them passes, and the only difference that I can see between the two is that the one that is passing previously passes legitimately...

Anyways, with OCMock setup correctly my tests are working as they should

Upvotes: 1

ansible
ansible

Reputation: 3579

Build Clean worked for me... sometimes anyways. Product -> Clean.

Upvotes: 2

Michal K.
Michal K.

Reputation: 2517

If you look at the output console, you should notice that Xcode starts running test suite:

Test Suite 'SomeClassTests' started at 2013-11-30 23:19:34 +0000

but suddenly exits without running test cases:

Program ended with exit code: 0

and shows a notification that test failed.

Restarting Xcode only helps temporarily as the problem quickly comes back.

I stopped executing tests in Xcode and run all tests by command line using xctool.

Upvotes: 1

willc2
willc2

Reputation: 39671

Until the bug is fixed, close the project and re-open it.

Upvotes: 2

squarefrog
squarefrog

Reputation: 4832

This is definitely a bug. Submit a report via https://bugreport.apple.com/ feel free to reference my report (15251050).

Upvotes: 17

Related Questions