CODEWITHSUNDEEP
javascriptseleniumtestingprotractorend-to-end
alecxe
alecxe

Reputation: 473873

Automatically detect test coupling in Protractor (randomizing test execution order)

The Problem:

We have a rather large test codebase. From time to time, instead of executing all the tests, we execute them individually or in packs. But, sometimes, we see the unexpected test failures because of the tests being interconnected, coupled. For example, one test assumes there is some data created by a previous test - running this kind of test individually will fail.

The Question:

Is it possible to automatically detect which Protractor tests are coupled in the project?

Our current idea is to somehow randomize the test execution order or randomly pick up a pack of tests from all the available tests and check if there are no failures. Hence, the other question: is it possible to change/randomize the Protractor test discovery and change the order of test execution?

Inspired by the Ned Batchelder's "Finding test coupling" blogpost and the Python nose test runner's nose-randomly plugin:

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google’s C++ test runner googletest.

Upvotes: 20

Views: 449

Answers (3)

Vishal Aggarwal
Vishal Aggarwal

Reputation: 4178

Did you try shuffling "it" blocks like below:

var shuffle = function (items) {
  var item, randomIndex;      
  for(var i = 0; i < items.length; i++){
    randomIndex= (Math.random() * items.length) | 0;
    item = items[i];
    items[i] = items[randomIndex];
    items[randomIndex] = item;
  }
}

describe('Suite', function() {

  it("should a", function () {
      console.log("execute a");
  });

  it("should b", function () {
      console.log("execute b");
  });

  it("should c", function () {
      console.log("execute c");
  });

  shuffle(this.children);    // shuffle the 'it' blocks

});

Source: Can protractor tests be run in a random order?

Upvotes: 6

Brine
Brine

Reputation: 3731

You can run tests randomly (at the file level) by setting the random property in your config . You can also set your salt/seed so it's reproducibly random. 

/**
 * If true, run specs in semi-random order
 */
random?: boolean,
/**
 * Set the randomization seed if randomization is turned on
 */
seed?: string,

You could also turn on shardTestFiles (parallel test runs), which should also be very telling in how coupled your tests are.

Upvotes: 13

JeffC
JeffC

Reputation: 25611

One problem is you likely have no idea how tests might be coupled. If one test referenced some variables from another test, you might be able to find those automatically but that's only one way tests might be coupled and probably not a likely scenario.

My first thought was to just run them individually and see which ones fail. The problem is that if you aren't cleaning state between tests, you might change the order (randomizing them, as you suggested) but if test 50 expects the data that test 20 set up but in the new order, test 20 still runs before test 50... test 50 will still pass. You will find some but probably not all until you run all of them in a random order several times.

You don't describe your application but my second thought was if there was a way to get back to a clean slate between tests, you should be able to find the tests that rely on other tests to set up data. I'm kinda surprised you aren't doing that already but if there's a long setup process that has to run to install a clean slate, etc. that might be an issue. Depending on your system, you might be able to snapshot a VM after a clean install and restore it to "quickly" get back to clean or you may be able to roll back SQL tables, etc. It really depends on your system and without more details on the system, it's hard to offer advice.

Another option is to go to those that wrote or maintain the tests and have them self-identify the tests they own that are coupled and fix them. This likely won't find them all but it might be a semi-quick start.

Oh... I just thought of one more thing. If you could reverse the order of test execution, that should be better than randomizing the execution. With reverse order, NO test would run after it's former predecessor and you should be able to find them all in one go.

Upvotes: 4

Related Questions