How can failing test cases be generated via EvoSuite？

Question

When I was learning to use Evosuite, I wrote some wrong methods. In one of them, for example, a method of class A directly calls an abstract method of interface B, and EvoSuite generated a mock for B to avoid the case failing:

import org.junit.Test;
import static org.junit.Assert.*;
import static org.evosuite.shaded.org.mockito.Mockito.*;
import MockTest.A;
import MockTest.B;
import org.evosuite.runtime.EvoRunner;
import org.evosuite.runtime.EvoRunnerParameters;
import org.evosuite.runtime.ViolatedAssumptionAnswer;
import org.junit.runner.RunWith;

@RunWith(EvoRunner.class) @EvoRunnerParameters(useVNET = true, separateClassLoader = true, useJEE = true) 
public class A_ESTest extends A_ESTest_scaffolding {

  @Test(timeout = 4000)
  public void test0()  throws Throwable  {
      A a0 = new A();
      B b0 = mock(B.class, new ViolatedAssumptionAnswer());
      doReturn((String) null).when(b0).get(anyInt());
      String string0 = a0.test(b0);
      assertNull(string0);
  }
}

For a method where a divide by zero may occur, the EvoSuite-generated test case even catches an exception to ensure that the test case passes:

import org.junit.Test;
import static org.junit.Assert.*;
import static org.evosuite.runtime.EvoAssertions.*;
import org.evosuite.runtime.EvoRunner;
import org.evosuite.runtime.EvoRunnerParameters;
import org.junit.runner.RunWith;
import test.Case2;

@RunWith(EvoRunner.class) @EvoRunnerParameters(useVNET = true, separateClassLoader = true, useJEE = true) 
public class Case2_ESTest extends Case2_ESTest_scaffolding {

  @Test(timeout = 4000)
  public void test8()  throws Throwable  {
      Case2 case2_0 = new Case2();
      // Undeclared exception!
      try { 
        case2_0.add(6, 0);
        fail("Expecting exception: ArithmeticException");

      } catch(ArithmeticException e) {
         //
         // / by zero
         //
         verifyException("test.Case2", e);
      }
  }
}

Are EvoSuite's automatically generated test cases always passing, as in these examples? How can EvoSuite be made to generate test cases that can fail, and thus can be directly used by developers?

Answer 1

Are EvoSuite's automatically generated test cases always passing, as in these examples?

The EvoSuite site provides an extensive list of publications describing the theory and operation of this software. Boiling it down to the simplest terms, however, the software generates test cases that detect mutations in the software under test that change its behavior and are not caught by existing tests. The actual behavior of the software under test is therefore the baseline, so none of the generated tests should fail on the unmodified software.

How can EvoSuite be made to generate test cases that can fail, and thus can be directly used by developers?

I don't see how that could be possible with any software. How is EvoSuite supposed to recognize any particular behavior of the software as incorrect, so as to generate a failing case? Consider your divide-by-zero example. The method under test evidently throws an ArithmeticException when its second argument is 0. How is EvoSuite supposed to know that that's not exactly what the method should do?

If the software could discern a priori what the code under test was supposed to do, then a better application of that remarkable feat would be to have it (re)write the actual code.

But that does not mean EvoSuite tests cannot be used by developers. Developers can use them in at least these ways:

1. Developers can analyze the generated tests to detect unexpected behaviors exhibited by the software under test. This is a pretty cool and valuable thing to do, but perhaps it doesn't qualify for you as using the tests "directly".

2. Many of the tests will correctly assert expected behavior, and these can be used as-is in the software's test suite. The generated overall suite will provide better protection against regressions than a lot of software has.

3. Even tests that assert incorrect behavior may be useful, not for detecting flaws in advance but for characterizing them after they are detected by other means. For example, suppose you get a report from the field that method widget.frob() threw an unexpected UpYoursException, but the circumstances that caused it are unclear. Chances are good that your EvoSuite tests already show at least one way to elicit this exception from the affected method. This may not qualify as using the tests "directly", but it is justification for not trying to weed out incorrect automatic test cases in advance (without also fixing the code under test).

Bottom Line: EvoSuite provides a complement to hand-written test cases, not a substitute for them.