Mockito fails with "Argument(s) are different" but debugger shows differently

Question

I am testing a method, which uses Mockito mocked objects. When I run the test, Mockito tells me that the parameters passed into the argument are different than expected. If I set a breakpoint within the method I am testing, the argument that it is called with is actually what I expect. This method only gets called once. Why is Mockito reporting something different from what I see with the debugger, and how can I fix this?

Here is the test:

@Test
public void addExclusivePermutationsTest() {
    Permutation p1 = mock(Permutation.class);

    PermutationBuilder pb = new PermutationBuilder();

    when(p1.applyPermutations(anySetOf(String.class)))
            .thenReturn(Collections.singleton("abc123"));

    pb.addExclusivePermutations(p1);
    pb.permute("test");

    verify(p1).applyPermutations(new HashSet<>(
            Collections.singletonList("test")));
}

The test is pretty simple. Permutation is an interface that contains the method definition Set<String> applyPermutations(Set<String>). When that method is called on p1, I tell Mockito to return a set containing [abc123].

The addExclusivePermutations(p1) call simply adds p1 to an ExclusivePermutation object that contains a list of Permutation objects. ExclusivePermutation is an implementor of Permutation so its applyPermutations method looks like this:

public Set<String> applyPermutations(final Set<String> stringPermutations) {
    Set<String> exclusivePermutations = new HashSet<>();

    for (Permutation permutation : permutationList) {
        exclusivePermutations.addAll(permutation.applyPermutations(stringPermutations));
    }

    return exclusivePermutations;
}

The above method is called inside permute("test"). The string "test" is converted into a Set, and passed into the above method. The above method is what calls applyPermutations on p1. When I look at this method with a debugger, stringPermutations contains only [test], as I expect it to, and is passed directly into p1. Then p1 returns [abc123] because it's mocked, and the above method joins it with stringPermutations to return [test, abc123]. So for some reason, Mockito is saying that stringPermutations is not [test] as the debugger shows, but instead what the method returns.

Finally, here is the error:

Argument(s) are different! Wanted:
permutation.applyPermutations(
    [test]
);
-> at PermutationBuilderTest$AddExclusivePermutationTests.addExclusivePermutationsTest(PermutationBuilderTest.java:87)
Actual invocation has different arguments:
permutation.applyPermutations(
    [abc123, test]
);

One last note (for those of you who are still reading). If I bypass ExclusivePermuation by copying ExclusivePermutation.applyPermutation() and putting it directly inside PermutationBuilder, the test passes. It is a mystery...

EDIT:

So I've narrowed it down a bit. In PermutationBuilder I have this method:

public Set<String> permute(String s) {
    for (Modifier modifier : modifierList) {
        s = modifier.applyModification(s);
    }

    Set<String> stringPermutations = new HashSet<>(Collections.singletonList(s));

    for (Permutation permutation : permutationList) {
        Set<String> result = permutation.applyPermutations(stringPermutations);
        stringPermutations.addAll(result);
    }

    return stringPermutations;
}

After I assign result, Mockito says that the method was called with [test]. However, when I step to the next line to add it to stringPermutations, it changes to [abc123, test]. You can see this in the two screenshots below (look at the bottom at the arguments array):

Answer 1

There are two issues opened for the same matter in the mockito github:

• Modifying invocation arguments in custom answer causes verification of those arguments fail #1153
• Capturing arguments when calling verify, not when real method executed #1287

So, first of all, creators consider it as a limitation of a framework as copying arguments at call-time would affect the performance and there is no common way to do a deep-copy anyway.

Secondly, the statement above explains what you should do in such situations: you need to clone arguments on your own. I haven't tried, but here are the suggested ways:

• org.mockito.internal.stubbing.answers.ClonesArguments
• Do a manual copy in the doAnswer and verify it instead of captor

However, when I used ClonesArguments for my use case:

doAnswer((invocation) -> {
    new ClonesArguments().answer(invocation);
    // in my case situation is different and instead of modifying invocation
    // my argument is the same collection 4 times with different elements
    // in your case it may or may not work
    // maybe you will have to copy paste and update their code
    return myAnswer;
})

I see in debug that elements of my collection are copied, but at the verify stage I still see only the most recent value, so didn't work for me.

Answer 2

The issue is that you're modifying the set that you're using as the argument.

That is, in the code

Set<String> stringPermutations = new HashSet<>(Collections.singletonList(s));

for (Permutation permutation : permutationList) {
    Set<String> result = permutation.applyPermutations(stringPermutations);
    stringPermutations.addAll(result);
}

You create the stringPermutations object, and add test to it. You now have a collection with one element.

You then call permutation.applyPermutations, and add the result of that to stringPermutations.

stringPermutations now contains test and abc123.

What is almost certainly going on here is that Mockito is not cloning the arguments (that would be... tricky to get right), and it is simply retaining a reference to the object(s) passed in.

Since you mutate stringPermutations after calling the method, Mockito thinks you called the method with a collection containing test and abc123, because that's what's in the collection when it attempts to verify the assertion. The simplest way to fix this is to not modify the collection after you've passed it into the method.

A solution could be:

Set<String> stringPermutations = new HashSet<>();

for (Permutation permutation : permutationList) {
    Set<String> result = permutation.applyPermutations(Collections.singletonList(s));
    stringPermutations.addAll(result);
}

stringPermutations.addAll(Collections.singletonList(s));

If you really need stringPermutations to contain both elements.