CODEWITHSUNDEEP
javaseleniumtestng
M_Tech
M_Tech

Reputation: 350

An Easier Way to Control Test Order in TestNG?

I've seen a few similar questions posted but none that quite answered my question directly... I'm using Selenium Webdriver 2.0 with TestNg and Java and I'm looking for a better way to control the order in which tests are run. Really all I want is the tests cases to run in the order they're written in the classes. It's intuitive. Its easy. Its maintainable. And I'm really surprised it's not the default for testNg. (Why apply some random ordering when it's not explicitly requested by the coder?)

Anyway, I know that I can add a list of methods to my testng xml, but I have 130 tests so far (and several hundred to come!) and such a method list will be far too much for future maintenance. EG when I add ten test methods I'll have to remember to add all those methods to my xml file. And what happens if I miss adding one? I'll never notice that it was never run...

So this solution would be a maintenance nightmare:

<test name="BVTQA01">
 <classes>
  <class name="TestPackage.BVT">
   <methods>
    <include name="testLogin"></include>
    <include name="testAddToCart"></include>
    ...
    <include name="test999 etc"></include>
   </methods>
  </class>
 </classes>
</test>

I've also tried preserve-order:

<test name="BVTQA01" preserve-order="true">
 <classes>
  <class name="TestPackage.TestBVT" />
 </classes>
</test>

but it's apparently ignored if I don't add a list of methods and the maintenance nightmare of 'include name' lists...

So currently I'm just listing my test classes in the xml file (as above- TestBVT which has 20 methods, etc.), and controlling test run order with 'depends on' annotations on the tests themselves. However this is not ideal as I am creating dependencies on every method. And I want to remove the dependencies where they are not really necessary. I only want to use 'depends on' when there is a real dependency.

I've also searched for creating the xml automatically from my @Test annotated methods. But the solutions posted are not clear on how to actually implement.

Any feedback on how to just make testNg run through my test classes from top to bottom, in order written, without any random sorting- or maintenance heavy list generation would be wonderful. Thanks in advance,

JR

Upvotes: 2

Views: 8333

Answers (3)

Pavel
Pavel

Reputation: 11

You can automatically generate priority annotation for each test method by implementing and registering IAnnotationTransformer, which reads method line number using javassist and assigns it as a TestNG test priority.

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import javassist.*

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public class AssignTestPriorityTransformer implements IAnnotationTransformer
{
    static ClassPool s_ClassPool = ClassPool.getDefault(); 

    @Override
    public void transform(ITestAnnotation p_annotation, Class p_testClass, Constructor p_testConstructor, Method p_testMethod)
    {
        p_annotation.setPriority(getMethodLineNumber(p_testMethod));
    }
    private int getMethodLineNumber(Method p_testMethod)
    {
        try
        {
            CtClass cc = s_ClassPool.get(p_testMethod.getDeclaringClass().getCanonicalName());
            CtMethod methodX = cc.getDeclaredMethod(p_testMethod.getName());
            return methodX.getMethodInfo().getLineNumber(0);        
        }
        catch(Exception e)
        {
            throw new RuntimeException("Getting of line number of method "+p_testMethod+" failed", e);
        }
    }
}

and register that listener in your XML file

<suite name="Suite" parallel="false">
<listeners>
    <listener class-name="AssignTestPriorityTransformer" />
</listeners>
<test name="All tests">
    <classes>
        <class name="tests.YourTests"/>
    </classes>
</test>
</suite>

The AssignTestPriorityTransformer will assure that methods marked by @Test annotation will be processed in same order like they are defined in the java source code of the test class.

Upvotes: 1

praneel
praneel

Reputation: 1902

One way is to run the TestNG programatically. 

package com.shn.test;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import javax.xml.parsers.ParserConfigurationException;

import org.testng.TestNG;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlTest;
import org.xml.sax.SAXException;

import com.shn.demos.RunDemo;

public class RunSuite{

    public static void main(String[] args)  {

        TestNG testNG = new TestNG();
        List<XmlSuite> suites = new ArrayList<XmlSuite>();

        //pass the name of Suite, Name of the groups to be executed & name of test
        suites.add(createSuite("SuiteDemo", "groupName1", "testName"));

        testNG.setSuiteThreadPoolSize(1);
        testNG.setXmlSuites(suites);

        testNG.run();
    }

    public static XmlSuite createSuite(String suiteName, String groupName, String testName) {
        XmlSuite suite = new XmlSuite();
        suite.setName(suiteName);
        suite.setParallel(XmlSuite.PARALLEL_NONE);

        LinkedHashMap<String, String> suiteParams = new LinkedHashMap<String, String>();
        //Put in the parameters out here which are required @ suite level
        suiteParams.put("SuiteKey1", "SuiteValue1");
        suiteParams.put("SuiteKey2", "SuiteValue2");
        suite.setParameters(suiteParams);

        XmlTest test = new XmlTest(suite);

        test.setName(testName);
        test.addIncludedGroup(groupName);
        //Put in the parametes out here wich are required @ test level
        test.addParameter("testKey1", "testValue1");
        test.addParameter("testKey2", "testValue2");

        List<XmlClass> clazzes = new ArrayList<XmlClass>();

        //This is your class under test
        XmlClass clazz = new XmlClass(Foo.class);
        clazzes.add(clazz);
        test.setClasses(clazzes);
        List<XmlTest> tests = new ArrayList<XmlTest>();
        tests.add(test);
        suite.setTests(tests);
        return suite;
    }
}

I am only aware of setPreserveOrder() havent really tried it though. Let me know if it works for you

Upvotes: 2

Evgheni Crujcov
Evgheni Crujcov

Reputation: 470

You can use priority annotations, they're not standard part of TestNG, but are made by the author of the framework as well. Of course, now instead of managing 1 xml file, you'll have to keep track of priorities for all of your 130 tests, but nevertheless you'll have a finer level of control using priorities. Priority annotations are described here: http://beust.com/weblog/2008/03/29/test-method-priorities-in-testng/

Upvotes: 0

Related Questions