Timing two testNG test methods to run in parallel and to run at EXACTLY the same time?

Question

I'm trying to test for possible race conditions on my site and so want two testNG test methods (using selenium 2.0) to execute in two different browsers at exactly the same time. I'm using the testNG plugin for Eclipse (Indigo). The timing is important because I want to see what happens when two updates are performed at the same time, and whether it will throw an exception. I've tried setting the 'parallel' attribute in the testng.xml to classes, to tests and to methods but one test always starts ahead of the other, so they are out of sync. Also one test may take longer to execute (by a few milliseconds) so I'm unsure how to get the timing just right.

For the moment I have the tests in two different classes as this gives me the closest result to what I want.

Class 1:

public class SyncTest {

WebDriver driver;
WebElement element;


@BeforeMethod
public void setUp() throws Exception {

    driver = new FirefoxDriver();

}


@Test(timeOut = 15000)
public void testSync() throws Exception {
    driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    //Login
    driver.get("http://mySite.com");
    driver.findElement(By.name("username")).sendKeys("admin");
    driver.findElement(By.name("password")).sendKeys("admin");
    try {
    element = driver.findElement(By.id("centralLogin_0"));
    element.click();
    } catch (Exception e){
        System.out.println("Error submitting: " + e.toString());
    }

    //open issue
    driver.findElement(By.linkText("Sync Test")).click();
    driver.switchTo().frame("mobile_vault");
    //Change status
    Select select = new Select(driver.findElement(By.id("statusSelect")));
    //select.deselectAll();
    select.selectByVisibleText("Open");
    List <WebElement> elements;
    //driver.findElement(By.className("button nextDialogue")).click();
    try {
    driver.findElement(By.cssSelector("div.button.nextDialogue > div")).click();
    } catch (Exception e){
        System.out.println("Not found");
    }

    Thread.sleep(2000);    
   try {
    driver.findElement(By.xpath("//form[@id='frmcall']/fieldset/div[14]/div[2]/div[1]")).click();
   } catch (Exception e) {
       System.out.println("Not Found");
       System.out.println(e.toString());
   }

   Thread.sleep(3000);
   driver.navigate().refresh();





}





@AfterMethod
public void tearDown() {
    driver.quit();
}

}

This is class 2:

public class NewSyncTest {

WebDriver driver;
WebElement element;

  @Test
  public void testF() throws Exception {

      driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

  //Login
  driver.get("http://mySite.com");
  driver.findElement(By.name("username")).sendKeys("admin");
  driver.findElement(By.name("password")).sendKeys("admin");

  try {
    element = driver.findElement(By.id("centralLogin_0"));
    element.click();
    } catch (Exception e){
        System.out.println("Error submitting: " + e.toString());
    }


    //open admin page
    driver.get("http://mySite.com/admin");

    Thread.sleep(2000);



try {

    driver.findElement(By.cssSelector("input[type='submit']")).click();

} catch (Exception e){
        System.out.println("Not found");
    }



  }
  @BeforeMethod
  public void beforeMethod() {

  driver = new FirefoxDriver();
  }

  @AfterMethod
  public void afterMethod() {

  driver.quit();
  }

}

I realise my use of testNG annotations is probably incorrect as I'm not sure which would be best (@BeforeSuite, @BeforeMethod etc.)

My testng.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="classes">
<test name="Test" preserve-order="false">
<classes>
  <class name="syncTest.SyncTest"/>
  <class name = "syncTest.NewSyncTest"/>


</classes>
</test>
</suite>

Like I said the tests never run exactly in parallel and I don't know how to achieve this :( I'll also include the testng-results.xml below, note the times of each test method:

<test name="Test" duration-ms="20264" started-at="2012-02-27T15:52:02Z" finished-    at="2012-02-27T15:52:23Z">
  <class name="syncTest.SyncTest">
    <test-method status="PASS" signature="setUp()[pri:0, instance:syncTest.SyncTest@3e9d75c5]" name="setUp" is-config="true" duration-ms="8475" started-at="2012-02-27T15:52:02Z" finished-at="2012-02-27T15:52:11Z">
    </test-method>
    <test-method status="PASS" signature="testSync()[pri:0, instance:syncTest.SyncTest@3e9d75c5]" name="testSync" duration-ms="11556" started-at="2012-02-27T15:52:11Z" finished-at="2012-02-27T15:52:22Z">
    </test-method>
    <test-method status="PASS" signature="tearDown()[pri:0, instance:syncTest.SyncTest@3e9d75c5]" name="tearDown" is-config="true" duration-ms="217" started-at="2012-02-27T15:52:22Z" finished-at="2012-02-27T15:52:23Z">
    </test-method>
  </class>
  <class name="syncTest.NewSyncTest">
    <test-method status="PASS" signature="beforeMethod()[pri:0, instance:syncTest.NewSyncTest@539ef8a0]" name="beforeMethod" is-config="true" duration-ms="4345" started-at="2012-02-27T15:52:02Z" finished-at="2012-02-27T15:52:07Z">
    </test-method>
    <test-method status="PASS" signature="testF()[pri:0, instance:syncTest.NewSyncTest@539ef8a0]" name="testF" duration-ms="9728" started-at="2012-02-27T15:52:07Z" finished-at="2012-02-27T15:52:16Z">
    </test-method>
    <test-method status="PASS" signature="afterMethod()[pri:0, instance:syncTest.NewSyncTest@539ef8a0]" name="afterMethod" is-config="true" duration-ms="231" started-at="2012-02-27T15:52:16Z" finished-at="2012-02-27T15:52:17Z">
    </test-method>
  </class>
</test>

Any and all suggestions are welcome. Thanks in advance! :)

Answer 1

Testing for race conditions cannot be deterministic just as High Performance Mark has noted. If you, on the other hand, want to test a single test method executing in parallel as multiple threads, then you can try something as shown below.

public class ConcurrentTestRunnerTest {

    @Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
    public void shouldRunInParallel1() {
        System.out.println("I'm running on thread " + Thread.currentThread().getName());
    }

    @Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
    public void shouldRunInParallel2() {
        System.out.println("I'm running on thread " + Thread.currentThread().getName());
    }

}

This will ensure that you have as many number of threads you configure are executed in parallel.

IMO, you should be able to achieve such results using some performance test tools like JMeter. Just mentioned this because you are using selenium.

Answer 2

Trying to test for race conditions by having two accesses happen at 'exactly the same time' is probably a futile exercise, and not useful either. Futile because you'll never achieve it, useless because data races don't arise when accesses occur at 'exactly the same time' but when the atomic operations (usually read and write) interleave in such a way that the views of the shared variables that the two processes get are inconsistent.

If you have 2 processes each doing, say, a read followed by a write from/to the same variable, think instead of all the ways in which the 4 atomic ops can occur (R1 - process 1 reads, etc):

R1,W1,R2,W2
R1,R2,W2,W1

etc.

Rewrite your tests so that you can control the intervals between R and W on each process, make those intervals long enough to control the interleaving of atomic operations, and check that your program copes.