Selenium and TestNG Using Both 'dependsOn' and 'priority =' Issue

Question

I am working on implementing better workflow control in my GUI automation tests. And I first started with dependsOn but quickly found the drawback is that if one test fails, the whole rest of the suite is not run. So I switched to using 'priority=' but am seeing unexpected behavior. One example:

@Test(priority = 10)
public void login(){...}

@Test(priority = 20, dependsOnMethods = "login")
public void verifyUserLogin() {...}

@Test(priority = 30, dependsOnMethods = "verifyUserLogin")
public void navigateToReportSettings() {...}

@Test(priority = 40, dependsOnMethods = "navigateToReportSettings")
public void verifyGeneralSettings() {...}

@Test(priority = 40, dependsOnMethods = "navigateToReportSettings")
public void verifyReportingPeriod() {...}
...
@Test(priority = 90, dependsOnMethods = "navigateToReportSettings")
public void saveReportSettings() {...}

What I want to happen:

1. Login.
2. Verify user is logged in.
3. Navigate to report settings page.
4. Verify the general settings and reporting period on the report settings page (in any order)
5. Make some changes and save.
6. Important: 10, 20, and 30 must succeed or skip the rest. If any 40 fails continue to 50 after all 40 have completed. But without dependency on any step 40s to succeed!

What is happening:

1. Login (priority 10).
2. Save (priority 90).

Note: there are also 'groups' annotations but I don't think that is relevant here. Thanks in advance for any tips on how to combine priority and dependsOn successfully to control workflow, but with dependencies only used where needed.

Here is another sample code. I have no idea why it is running in this order: Output: 10, 20, 30, 40, etc... 110, //OK 130, 140, 150, 160, // Why were the 120 priorities skipped? 120, 120, 120, etc... 120 //Run last? Also interesting is that the group of 120s can be renumbered sequentially (121, 122, 123, etc) and still they are run last.

Therefore the issue must be that 'dependsOn' and 'priority =' simply do not play well together. And I'm curious if anyone has gotten these two working in their environment. Who knows maybe its a but with Intellij IDEA? Anyway I need to get to the bottom of this soon to avoid costly refactoring later! Thanks again for any feedback- JR

@Test(priority = 10, groups = "A")
public void login(){

System.out.println("10");

}


@Test(priority = 20, groups = {"A", "B"})
public void openUserAdministrationTest() {

    System.out.println("20");

}

@Test(priority = 30, groups = {"A", "B"})
public void usersTabTest() {

    System.out.println("30");

}

@Test(priority = 40, groups = {"A", "B"})
public void createUserTabTest() {

    System.out.println("40");

}


@Test(priority = 50, groups = {"A", "B"})
public void userCreationDataEntryTest() {

    System.out.println("50");

}

@Test(priority = 60, groups = {"A", "B", "C"})
public void userRolesTest() {

    System.out.println("60");

}

@Test(priority = 70, groups = {"A", "B"})
public void saveUserTest() {

    System.out.println("70");

}

@Test(priority = 80, groups = {"A", "B"})
public void closeUserAdminAndLogoutTest() {

    System.out.println("80");

}

@Test(priority = 90, groups = "A")
public void loginNavigateToUserAdmin() {

    System.out.println("90");
}

@Test(priority = 100, groups = {"A", "D"})
public void verifyUserSearchUserReturned() {

    System.out.println("100");

}

@Test(priority = 110, groups = {"A", "D"})
public void reOpenNewUserTest() {

    System.out.println("110");

}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserUserNameTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserFullNameTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserDepartmentTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserPhoneNumberTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserEmailTest() {

    System.out.println("120");
}

//      Note: password and active verified by user login

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserActiveCheckedTest() {
    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserLanguageTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserDateFormatTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserNumberFormatTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserReportingPeriodTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserReportingPeriodExampleTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserReferencePeriodTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserReferencePeriodExampleTest() {

    System.out.println("120");
}

@Test(priority = 120, groups = {"A", "E"}, dependsOnMethods = "reOpenNewUserTest")
public void verifyNewUserShowAnnotationsCheckedTest() {
    System.out.println("120");
}

@Test(priority = 130, groups = {"A", "C"})
public void verifyNewUserRoleTest() {

    System.out.println("130");
}


@Test(priority = 140, groups = {"A", "C"})
public void verifyNewUserFunctionalRoleTest() {

    System.out.println("140");

}

@Test(priority = 150, groups = {"A", "C"})
public void verifyUserAdminCloseAndLogoutTest() {

    System.out.println("150");

}

@Test(priority = 160, groups = {"A", "C"})
public void verifyUserLogin() {

    System.out.println("160");

}

This is a much simpler example, but also showing how depends on simply breaks priorities:

@Test(priority = 10)
public void test10(){
    System.out.println("10");
}

@Test(priority = 20, dependsOnMethods = "test10")
public void test20() {
    System.out.println("20, depends on 10");
}

@Test(priority = 30, dependsOnMethods = "test20")
public void test30() {
    System.out.println("30, depends on 20");
}

@Test(priority = 40, dependsOnMethods = "test10")
public void test40() {
    System.out.println("40, depends on 10");
}

Should run: 10, 20, 30, 40. Runs: 10, 20, 40, 30.

Answer 1

To make tests run even after it fails, use alwaysRun attribute along with dependsOnMethods instead of using priority attribute, alwaysRun attribute let the next dependency to execute even after it fails,try the following syntax:

@Test
public void login(){...}

@Test(dependsOnMethods = "login")
public void verifyUserLogin() {...}

@Test(dependsOnMethods = "verifyUserLogin")
public void navigateToReportSettings() {...}

@Test(dependsOnMethods = "navigateToReportSettings", alwaysRun=true)
public void verifyGeneralSettings() {...}

@Test(dependsOnMethods = "navigateToReportSettings", alwaysRun=true)
public void verifyReportingPeriod() {...}
...
@Test(dependsOnMethods = "navigateToReportSettings")
public void saveReportSettings() {...}

Answer 2

Don't provide priority and depends on together, you can group the tests. You can do it like For example,

@Test(priority = 10, groups = { "10" })
public void test10() {
    System.out.println("10");
}

@Test(dependsOnMethods = "test10", groups = { "10" })
public void test20() {
    System.out.println("20, depends on 10");
}

@Test(dependsOnGroups = { "10" })
public void test30() {
    System.out.println("30, depends on 20");
}

@Test(dependsOnMethods = "test30")
public void test40() {
    System.out.println("40, depends on 10");
}

Second thing for must run (succeed or skip the rest)

You will always be run after the methods you depend on, even if some of them have failed. This is useful when you just want to make sure that your test methods are run in a certain order but their success doesn't really depend on the success of others. A soft dependency is obtained by adding "alwaysRun=true" in your @Test annotation.

If a method depended upon fails and you have a hard dependency on it (alwaysRun=false, which is the default), the methods that depend on it are not marked as FAIL but as SKIP. Skipped methods will be reported as such in the final report (in a color that is neither red nor green in HTML), which is important since skipped methods are not necessarily failures.

Answer 3

There are a lot of defects related to dependsOnMethods annotation. I've faced similar situation in 6.5.2. Try to update your dependency to 6.8.3.