Automated Test case Execution - when to stop

Question

We have around 100 test cases for our system. We are trying to build an automated test suite for it.

Say while running the tests the 25th test fails. Should our automated test system bail out here and stop execution, or should it just mark this as failed and continue trying to execute test cases 26th onwards (that is every test cycle will execute all 100 test cases irrespective of any failed test cases).

Ofcourse after a failed test case(for example no 25) if the system needs to be reset to execute test cases 26 onwards it will be taken care of.

Thanks

James

Answer 1

If this is running without human intervention, say as part of some automated build, I would want to attempt all tests.

However, there are scenarios where you're in the mode of fixing problems where it might save a human's time to just stop. If it's easy I'd like to offer a "Stop on first failure" option.

Answer 2

If your tests are independent - you should finish all of them. This way you can monitor the system stability and see all of the problems at once without re-running tests countless times.