Customizing failure reporting in TestNG

Question

Background:

I have created a basic playground project that contains:

1. A testLogin.java file that contains:
   a. testng package imports (org.testng.*)
   b. selenium webdriver imports (org.openqa.selenium.*)
   c. 5 test-methods with testng annotations:

1. @Test(groups={"init"}) public void openURL()
   Contains webdriver code to initiate the webdriver and open a chrome >instance with a given url.
   

2. @Test(dependsOnGroups={"init"}) public void testLogin()
   Contains webdriver code to:
   1. Locate username password text-input elements, enter the username password from a properties file.
   2. Locate the "log in" button and click the button to log-in
   3. Manage a login-forcefully scenario if someone else has already logged in using the credentials.
   

3. @Test(dependsOnMethods={"testLogin"}) public void testPatientsScheduleList()
   Contains webdriver code to check if any patients have been scheduled. If yes, then fetch the names and display in console.
   

4. @Test() public void testLogout()
   Contains webdriver code to locate the logout button and click on the button to logout of the app.
   

5. @AfterTest() public void closeConnection()
   Contains webdriver code to dispose the webdriver object and close the chrome instance.

Currently I am simply running the test script wrapped as testng methods from ANT and a testng-xslt report gets generated.

Issues:

1. Performing validations against every line of code of webdriver script in a test method:

I know:
1. Selenium webdriver script contains API methods (findElement() and others.) that throw exceptions as a result of a default assertion/validation they perform. These exceptions show up in the generated report when a test-method fails.
2. TestNG provides Assert class that has many assertion methods but I have not yet figured out how can i use them to perform validation/assertions against every line of code of webdriver script. I tried adding assertion methods after every line of webdriver script code. What appeared in the output was just an AssertionError exception for a testmethod.

2. Failing a certain test method which gets passed due to try.. catch block.
If I use a try catch block around a set of 2 or more test drive script steps, and if a test-case fails in any of the steps (script line) then the try..catch block handles it thereby showing the test-method as "passed" in the execution report, which actually failed.

3. Creating a custom report which will show desired test execution results and not stack-traces!
When I execute the above script, a testng-xslt report gets generated that contains pass/fail status of each test method in a test-suite (configured in testng.xml). The test-results only give me whether a test-method has passed or failed and provides an exception's stack-trace which really doesn't provide any helpful information. I don't want such abstract level of test execution results but something like:

Name | Started | Duration | What-really-went-wrong (Failure)

Can anyone please suggest/ give some pointers regarding:
1. How can I perform validation/assertion against every line of code of webdriver script in a test-method without writing asserts after every script line?
2. How can I fail a certain test method which gets passed due to try catch block?
3. How can I customize the failure reporting so that I can send a failure result like "Expected element "button" with id "bnt12" but did not find the element at step 3 of test-method" to testng's reporting utility?
4. In the testng-xslt report I want to display where exactly in the test-method a failure occurred. So for example if my test-method fails because of a webelement = driver.findElement() at line 3 of a test-method, I want to display this issue in the test-report in the "What-really-went-wrong" column. I have read about testng testlisteners TestListenerAdapter / ITestListener/ IReporter but I don't understand how to use them after checking testng's javadocs.
5. Also, I have to implement PageObject pattern once I am done with customizing the test report. Where would be the right place to perform assertions in a page-object pattern? Should assertions be written in the page object test methods or in the higher level test methods that will use the PageObject classes?

P.S: I am completely new to testng framework and webdriver scripting. Please bear with any technical mistakes or observation errors if any in the post.

Answer 1

How can I customize the failure reporting so that I can send a failure result like "Expected element "button" with id "bnt12" but did not find the element at step 3 of test-method" to testng's reporting utility?

You can use extent report and testng listener class( in this class use onTestFailure method to customize your failure report).

Answer 2

How can I perform validation/assertion against every line of code of webdriver script in a test-method without writing asserts after every script line?

I dont think so. It is the assertions that does the comparison. So u need it.

How can I fail a certain test method which gets passed due to try catch block?

try-catch will mask the assertion failure.(because on assertion failure, an assertion exception is thrown, so if your catch block is like (catch(Exception e)) Assertion failures wont escape the catch block.

How can I customize the failure reporting so that I can send a failure result like "Expected element "button" with id "bnt12" but did not find the element at step 3 of test-method" to testng's reporting utility?

You need to use test listeners . TestNG TestListenerAdapter is a good start

Also, I have to implement PageObject pattern once I am done with customizing the test report. Where would be the right place to perform assertions in a page-object pattern? Should assertions be written in the page object test methods or in the higher level test methods that will use the PageObject classes?

My personal choice is to use assertions in Test methods, since it is where we are doing the actual testing. Page objects contains scripts for navigating inside the web page.