When spying on a CompletableFuture with Mockito, spyObj.get occasionally fails

Question

I'm running into a problem where occasionally my example code below will fail when running the test suite, but the tests individually always seem to pass. If I use just .get() for the spy CompletableFuture without specifying a timeout, it hangs indefinitely.

This problem occurs both on Windows, OS X, and I've tried a few different versions of the the Java 8 JDK.

I have this problem with Mockito 2.18.3 and Mockito 1.10.19.

I can run the example test suite code below sometimes 7-10 times successfully, but almost always when trying more than 10 times I'll see random test failures.

Any help would be greatly appreciated. I have also posted on the Mockito mailing list, but things look fairly quite there.

package example;


import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
import org.junit.Test;
import static org.mockito.Mockito.spy;


public class MockitoCompletableFuture1Test {

    @Test
    public void test1() throws Exception {

        CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> "ABC");
        CompletableFuture<String> futureSpy = spy(future);

        try {
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS));
        } catch (TimeoutException e) {
            assertEquals("ABC", future.get(1, TimeUnit.SECONDS));    // PASSES
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS)); // OCCASIONALLY FAILS
            fail("futureSpy.get(...) timed out");
        }

    }

    @Test
    public void test2() throws Exception {

        CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> "ABC");
        CompletableFuture<String> futureSpy = spy(future);

        try {
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS));
        } catch (TimeoutException e) {
            assertEquals("ABC", future.get(1, TimeUnit.SECONDS));    // PASSES
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS)); // OCCASIONALLY FAILS
            fail("futureSpy.get(...) timed out");
        }

    }

    @Test
    public void test3() throws Exception {

        CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> "ABC");
        CompletableFuture<String> futureSpy = spy(future);

        try {
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS));
        } catch (TimeoutException e) {
            assertEquals("ABC", future.get(1, TimeUnit.SECONDS));    // PASSES
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS)); // OCCASIONALLY FAILS
            fail("futureSpy.get(...) timed out");
        }

    }

    @Test
    public void test4() throws Exception {

        CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> "ABC");
        CompletableFuture<String> futureSpy = spy(future);

        try {
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS));
        } catch (TimeoutException e) {
            assertEquals("ABC", future.get(1, TimeUnit.SECONDS));    // PASSES
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS)); // OCCASIONALLY FAILS
            fail("futureSpy.get(...) timed out");
        }

    }

    @Test
    public void test5() throws Exception {

        CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> "ABC");
        CompletableFuture<String> futureSpy = spy(future);

        try {
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS));
        } catch (TimeoutException e) {
            assertEquals("ABC", future.get(1, TimeUnit.SECONDS));    // PASSES
            assertEquals("ABC", futureSpy.get(1, TimeUnit.SECONDS)); // OCCASIONALLY FAILS
            fail("futureSpy.get(...) timed out");
        }

    }

}

Answer 1

According to Important gotcha on spying real objects!:

Mockito *does not* delegate calls to the passed real instance, instead it actually creates a copy of it. So if you keep the real instance and interact with it, don't expect the spied to be aware of those interaction and their effect on real instance state. […]

So basically, it will take the state of your future at the time you call spy() on it. If it is already completed, then the resulting spy will be too. Otherwise, your spy will remain uncompleted, except if you complete it yourself.

As the asynchronous completion will be performed on the original future and not on your spy, it will thus not be reflected in your spy.

The only case where this would work properly, is when you have full control over it. This means you would have created your CompletableFuture with new, wrap it in a spy, and then only use that spy.

In general however, I would advise to avoid mocking futures, as you often don't have control over how they are handled. And as stated in Mockito's Remember section:

Do not mock types you don’t own

CompletableFuture is not a type you own.

Anyway, it shouldn't be necessary to mock CompletableFuture methods, as you can control what they do based on complete() or completeExecptionally(). On the other side, it shouldn't be necessary to check whether its methods are called since:

• the methods with side-effects (like complete()) can easily be asserted afterwards;
• the other methods are returning values that should make your test fail if they are not used.

Basically, CompletableFuture behaves similarly to a value object, and the documentation states:

Don’t mock value objects

If you feel your test cannot be written without using a spy, try to reduce it to an MCVE and post a separate question on how to do it.

Answer 2

When future is created (calling CompletableFuture.supplyAsync), it will also create a Thread (ForkJoinPool.commonPool-worker-N) to execute the lambda expression. That thread has a reference to the newly created object (future in our case). When the async job is finished, the thread (ForkJoinPool.commonPool-worker-N) will notify (wake up) the other thread (main) waiting for it that it has finished.

How does it know which thread is waiting for it? When you call the get() method the current thread will be saved as a field in the class and the thread will park (sleep) and will wait to be unparked by some other thread.

The problem is that futureSpy will save in its own field the current thread (main), but the async thread will try to read the information from the future object (null).

The problem doesn't always appear in your test case because if the async function is already finished, get won't put the main thread to sleep.

---

Reduced example

For testing purposes I've reduced you test cases to something shorter that reliably reproduces the error (except the first run):

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

import static org.mockito.Mockito.spy;

public class App {
   public static void main(String[] args) throws InterruptedException, ExecutionException, TimeoutException {
      for (int i = 0; i < 100; i++) {
         CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {
            try {
               Thread.sleep(500);
            } catch (InterruptedException e) {
               throw new RuntimeException(e);
            }
            return "ABC";
         });
         CompletableFuture<String> futureSpy = spy(future);
         try {
            futureSpy.get(2, TimeUnit.SECONDS);
            System.out.println("i = " + i);
         } catch (TimeoutException ex) {
            System.out.println("i = " + i + " FAIL");
         }
      }
   }
}

In my tests the output is:

i = 0
i = 1 FAIL
i = 2 FAIL
i = 3 FAIL