Stopping Test's if JUnit Test fails

Question

I'm trying to figure out How i can make my JUnits tests run a specific order as well as stop all test's if one of the test's fail. Here below i have included just some simple actionbar item test's. All i basically want to make sure is when i run my build/test via command line, i want the test to stop if one of these tests fails. Any Recommendations?

public void testPreconditions() {
    assertNotNull(instrumentation);
    assertNotNull(mWeb);
    assertNotNull(mActivity);
}

protected void setUp() throws Exception {
    super.setUp();

    instrumentation = getInstrumentation();
    mActivity = getActivity();
    setActivityInitialTouchMode(false);
    mWeb = (WebView) mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.webview_main);
    mMockWebViewClient = new MockWebViewClient();
    mWeb.setWebViewClient(mMockWebViewClient);
    mSettings = mWeb.getSettings();
}

public void testThatButtonReloadWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_refresh);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

public void testThatButtonForwardWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_forward);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

public void testThatButtonBackWorks() {
    final View view = mActivity
            .findViewById(com.jaisonbrooks.webview.R.id.menu_back);
    mActivity.runOnUiThread(new Runnable() {

        @Override
        public void run() {
            view.requestFocus();
            view.callOnClick();
        }
    });
}

Answer 1

Changing the tests order in jUnit5 can be done by using @MethodOrder

If you want to have a specific order, you should annotate your class to use OrderAnnotion

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderInJunit {

    @Test
    @Order(1)
    public void runsFirst() {
        System.out.println("runs first");
    }
    @Test
    @Order(2)
    public void runsSecond() {
        System.out.println("runs second");
    }

}

If you want that one test fails consecutive tests in vanilla junit, you can do something like this:

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
// One test instance per class, to keep the state of "previousTestFailed" between each test
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class OrderInJunit {

    private Boolean previousTestFailed = false;

    @Test
    @Order(1)
    public void runsFirst() {
        previousTestFailed = true;
        System.out.println("runs first");
        previousTestFailed = false;
    }

    @Test
    @Order(2)
    public void runsSecond() {
        Assumptions.assumeFalse(previousTestFailed,"No use to run this when \"runsFirst\" fails");
        previousTestFailed = true;
        System.out.println("runs second");
        fail("Test failure");
        previousTestFailed = false;
    }
    @Test
    @Order(3)
    public void runsThird() {
        Assumptions.assumeFalse(previousTestFailed,"No use to run this when \"runsSecond\" fails");
        previousTestFailed = true;
        System.out.println("runs second");
        previousTestFailed = false;
    }
}

This will always result in:

However, it's clear that this setup has some issues:

1. you change the order which you regulary don't want to do in jUnit
2. you create an instance per class instead of per test method
3. it does come with boilerplate since you have to flip the previousTestFailed boolean and you have to add the assumption in each test (except the first).

You probably want to look into a more E2E specific test framework or jUnit plugin(s).

Answer 2

I Haven't done Junit test for Android Apps, but based on my experience with web application, I am suggesting this answer.

When ever I face such situation, I'll call them in separate method with return type

Example:

public void testCheckFunctions()
{
boolean returnval=false;

returnval=test1();
if(returnval)
{
returnval=test2();
}
if(returnval)
{
returnval=test3();
}
}

You can exit the test if any of the above condition fails.

Answer 3

If you're doing integration tests using JUnit, you could prefix the method name with _1, _2 etc.:

public void _1_testThatButton()...
public void _2_testSomethingSecond()...

I suppose you could call System.exit() instead of throwing an exception, but if your tests are taking too long, that may be a bad sign for adding more tests in the future (adding more tests may discourage you from running them).

If your're using ant, you can try this: How to make junit testing to stop after first failing test