Testng 6.9.10 assertEquals() not failing like it should

Question

So I set up a Test case using TestNG. When I do an assert equals on something that should fail, the test passes anyway. I have tried comparing them as list and string. The actual comes out as:

actual = [The license key was not valid., 
          The license key has expired and is no longer valid., 
          That email domain is not allowed., 
          The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., 
          That license key does not have a valid start date, the administrator must define an effective date for the license key.]

The code that I am using is below. Actual and expected are clearly different but the test still passes. I am using TestNG 6.9.10. I haven't been able to find anywhere that there is a bug with assertEquals.

import org.testng.annotations.*;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;

@Test
public void verifyLicencingErrorsTest() {
    List<String> expected = Arrays.asList(
            "The license key was not valid.",
            "The license key has expired and is no longer valid.",
            "The license key was not valid.",
            "The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased.",
            "That license key does not have a valid start date, the administrator must define an effective date for the license key."
    );
    List<String> actual = createAccountPage.verifyLicencingErrors();

    try {
        assertEquals(actual.toString(), expected.toString(), "The actual alerts did not match what was expected");
    } catch (Error e) {
        verificationErrors.append(e.toString());
    }
}

I think I have provide all of the information necessary information. If I need more please let me know.

Edit: Added imports

Answer 1

As per the TestNG Assert documentation:

public static void assertEquals(java.lang.String actual,
                                java.lang.String expected,
                                java.lang.String message)

Asserts that two Strings are equal. If they are not, an AssertionError, with the given message, is thrown.
Parameters:
actual - the actual value
expected - the expected value
message - the assertion error message

The issue is that you're catching Error, thus the thrown AssertionError no longer reaches the framework and the test is considered ok. If you log the error you catch you can easily see that:

    try {
        assertEquals(actual.toString(), expected.toString(), "The actual alerts did not match what was expected");
    } catch (Error e) {
        log.error("Caught an [" + e.getClass().getName() + "]", e);
    }

... yelds:

[ERROR] 2016-03-30 12:51:01,345 test.LauncherTest - Caught an [java.lang.AssertionError]
java.lang.AssertionError: The actual alerts did not match what was expected expected [[The license key was not valid., The license key has expired and is no longer valid., The license key was not valid., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key.]] but found [[The license key was not valid., The license key has expired and is no longer valid., That email domain is not allowed., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key]]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:496)
    at org.testng.Assert.assertEquals(Assert.java:125)
    at org.testng.Assert.assertEquals(Assert.java:178)
    at test.LauncherTest.verifyLicencingErrorsTest(LauncherTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:821)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    at org.testng.TestRunner.privateRun(TestRunner.java:773)
    at org.testng.TestRunner.run(TestRunner.java:623)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
    at org.testng.SuiteRunner.run(SuiteRunner.java:259)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
    at org.testng.TestNG.run(TestNG.java:1018)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:125)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

While removing the catch-block, or potentially refining the exceptions caught, will work as expected:

java.lang.AssertionError: The actual alerts did not match what was expected 
Expected :[The license key was not valid., The license key has expired and is no longer valid., The license key was not valid., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key.]
Actual   :[The license key was not valid., The license key has expired and is no longer valid., That email domain is not allowed., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key]
  <Click to see difference>


    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:496)
    at org.testng.Assert.assertEquals(Assert.java:125)
    at org.testng.Assert.assertEquals(Assert.java:178)
    at test.LauncherTest.verifyLicencingErrorsTest(LauncherTest.java:41)

===============================================
Custom suite
Total tests run: 1, Failures: 1, Skips: 0
===============================================