How can I pass Methods as input to a @Test using a @DataProvider?

Question

I've been writing tests that inspect the @DataProviders for failures (since those tests get skipped silently) by getting a list of all the @DataProvider Methods in my package and running them. This works great, but when I tried implementing it using a @DataProvider (very Meta, I know) I ran into what seems to be a bug in TestNG. Of the following 4 cases, the only one that works is the encapsulated one:

package mypackage;

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

public class TestDataProvidersWithMethod {

    //////////////// Plain example: Assertion fails

    @DataProvider(name = "TestThisDP")
    public Object[][] testThisDP() throws IllegalAccessException, InstantiationException, InvocationTargetException, NoSuchMethodException {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP", (Class<?>[]) null);

        return new Object[][]{new Object[] {method}};
    }


    @Test(dataProvider = "TestThisDP")
    public void testGiveMethod(Method method) throws IllegalAccessException, InstantiationException, InvocationTargetException {
        System.err.println("Method: " + method.getName());

        Assert.assertTrue(method.getName().equals("testThisDP")); // FAILS: name is actually "testGiveMethod" for some reason.
    }

    /////// Encapsulated example, this works, but has extra fluff


    class Container{
        public Method method;
        public Class clazz;

        public Container(Method method, Class clazz) {
            this.method = method;
            this.clazz = clazz;
        }
    }    

    @DataProvider(name = "TestThisDP4")
    public Object[][] testThisDP4() throws IllegalAccessException, InstantiationException, InvocationTargetException, NoSuchMethodException {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP", (Class<?>[]) null);

        return new Object[][]{new Object[] {new Container(method,null)}};
    }

    @Test(dataProvider = "TestThisDP4")
    public void testGiveMethod(Container container) throws IllegalAccessException, InstantiationException, InvocationTargetException {
        System.err.println("Method: " + container.method.getName());

        Assert.assertTrue(container.method.getName().equals("testThisDP")); // Succeeds!!
    }


    /////////////////// Weird failure, test isn't run due to TypeMismatch

    @DataProvider(name = "TestThisDP2")
    public Object[][] testThisDP2() throws IllegalAccessException, InstantiationException, InvocationTargetException, NoSuchMethodException {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP2", (Class<?>[]) null);

        return new Object[][]{new Object[] {method ,""}};
    }


    @Test(dataProvider = "TestThisDP2")
    public void testGiveMethod2(Method method, String unused) throws IllegalAccessException, InstantiationException, InvocationTargetException {
        System.err.println("Method: " + method.getName());

        Assert.assertTrue(method.getName().equals("testThisDP")); // FAILS hard: Type mismatch!!!
    }

    /////////////////////// Attempt at understanding the failure above..
    /////////////////////// This fails like the plain example, from the assertion

    @DataProvider(name = "TestThisDP3")
    public Object[][] testThisDP3() throws IllegalAccessException, InstantiationException, InvocationTargetException, NoSuchMethodException {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP3", (Class<?>[]) null);

        return new Object[][]{new Object[] {"", method }};
    }


    @Test(dataProvider = "TestThisDP3")
    public void testGiveMethod3(String unused, Method method) throws IllegalAccessException, InstantiationException, InvocationTargetException {
        System.err.println("Method: " + method.getName());

        Assert.assertTrue(method.getName().equals("testThisDP")); // FAILS: name is actually "testGiveMethod" for some reason.
    }
}

Am I doing something wrong, or is this a bug in TestNG?

Answer 1

There's no bug here. When you basically have Method as one of the parameters of your @Test method, TestNG is basically trying to resort to doing something called as Native Injection.

So in your case, TestNG is basically injecting a Method reference that represents the currently "being invoked" @Test method.

To disable this Native Injection you would need to use the annotation @NoInjection.

Here's a fixed version of your same test code

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.NoInjection;
import org.testng.annotations.Test;

import java.lang.reflect.Method;

public class TestDataProvidersWithMethod {

    //////////////// Plain example: Assertion fails

    @DataProvider(name = "TestThisDP")
    public Object[][] testThisDP() throws Exception {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP", (Class<?>[]) null);

        return new Object[][]{new Object[]{method}};
    }

    @Test(dataProvider = "TestThisDP")
    public void testGiveMethod(@NoInjection Method method) {
        System.err.println("Method: " + method.getName());
        // FAILS: name is actually "testGiveMethod" for some reason.
        Assert.assertEquals(method.getName(), "testThisDP");
    }

    /////// Encapsulated example, this works, but has extra fluff
    class Container {
        public Method method;
        public Class clazz;

        Container(Method method, Class clazz) {
            this.method = method;
            this.clazz = clazz;
        }

        @Override
        public String toString() {
            if (clazz == null) {
                return method.getName() + "()";
            }
            return clazz.getName() + "." + method.getName() + "()";
        }
    }

    @DataProvider(name = "TestThisDP4")
    public Object[][] testThisDP4() throws Exception {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP", (Class<?>[]) null);
        return new Object[][]{new Object[]{new Container(method, null)}};
    }

    @Test(dataProvider = "TestThisDP4")
    public void testGiveMethod(Container container) {
        System.err.println("Method: " + container.method.getName());
        // Succeeds!!
        Assert.assertEquals(container.method.getName(), "testThisDP");
    }


    /////////////////// Weird failure, test isn't run due to TypeMismatch

    @DataProvider(name = "TestThisDP2")
    public Object[][] testThisDP2() throws Exception {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP2", (Class<?>[]) null);
        return new Object[][]{new Object[]{method, ""}};
    }


    @Test(dataProvider = "TestThisDP2")
    public void testGiveMethod2(@NoInjection Method method, String unused) {
        System.err.println("Method: " + method.getName());
        // FAILS hard: Type mismatch!!!
        Assert.assertEquals(method.getName(), "testThisDP2");
    }

    /////////////////////// Attempt at understanding the failure above..
    /////////////////////// This fails like the plain example, from the assertion

    @DataProvider(name = "TestThisDP3")
    public Object[][] testThisDP3() throws Exception {
        Class<?> aclass = this.getClass();
        Method method = aclass.getMethod("testThisDP3", (Class<?>[]) null);
        return new Object[][]{new Object[]{"", method}};
    }

    @Test(dataProvider = "TestThisDP3")
    public void testGiveMethod3(String unused, @NoInjection Method method) {
        System.err.println("Method: " + method.getName());
        // FAILS: name is actually "testGiveMethod" for some reason.
        Assert.assertEquals(method.getName(), "testThisDP3");
    }

}

For scenarios that doesn't involve @DataProviders you can refer to this link to understand the valid combinations of Native Injection.