CODEWITHSUNDEEP
javascriptunit-testingtestingqunitsaucelabs
nelsonic
nelsonic

Reputation: 33154

Why does SauceLabs say my QUnit Test Fails when its clearly passing?

We are trying to use SauceLabs to verify that our browser-based QUnit tests are passing in popular device/browser combinations ...

The tests PASS when we look at them in the browser: https://ordenado.herokuapp.com/

But for some reason SauceLabs is telling us they "failed"...

saucelabs-ordem-test-is-clearly-passing

See: https://saucelabs.com/tests/5b0f07813a7f4934bb44b07606ea2fd5

cURL Command Used

For reference, we used the following curl command:

curl https://saucelabs.com/rest/v1/ordem/js-tests \
-X POST \
-u ordem:SECRET_KEY \
-H 'Content-Type: application/json' \
--data '{
    "platforms": [
      ["Windows 8.1", "internet explorer", "11"],
      ["Windows 8", "internet explorer", "10"],
      ["Windows 8.1", "firefox", "beta"],
      ["Windows 8", "firefox", "37"],
      ["Windows 7", "firefox", "32"],
      ["OS X 10.8", "safari", "6"],
      ["OS X 10.8", "chrome", "37"],
      ["Linux", "chrome", "30"],
      ["Linux", "firefox", "dev"],
      ["OS X 10.10","iphone", "7.0"],
      ["OS X 10.10","iphone", "8.2"],
      ["OS X 10.10","ipad", "7.0"],
      ["OS X 10.10","ipad", "8.2"]
    ],
    "url": "https://qunit.herokuapp.com/test/test.html?coverage=true",
    "framework": "qunit",
    "name":"ordem",
    "public": "public",
    "build": "build-007"
}'

Also, does anyone else ever experience the following error:

"The Sauce VMs failed to start the browser or device"

saucelabs-vm-failed

Visit: https://saucelabs.com/u/ordem for complete list of tests. click on any of the ones Sauce claims "failed" and watch the video to see the tests passing!!

saucelabs-ordem-test-summary-fails

Any insight much appreciated!

Upvotes: 3

Views: 281

Answers (1)

just-boris
just-boris

Reputation: 9746

You should add several hooks to report qunit test results from SauceLabs.

There is an article how to get started with qunit. There mentioned repository with example where is following code snippet:

var log = [];
QUnit.done = function (test_results) {
  var tests = log.map(function(details){
    return {
      name: details.name,
      result: details.result,
      expected: details.expected,
      actual: details.actual,
      source: details.source
    }
  });
  test_results.tests = tests;

  // delaying results a bit cause in real-world
  // scenario you won't get them immediately
  setTimeout(function () { window.global_test_results = test_results; }, 2000);
};
QUnit.testStart(function(testDetails){
  QUnit.log = function(details){
    if (!details.result) {
     details.name = testDetails.name;
     log.push(details);
    }
 }
});

Add this code before your tests and results will be reported properly

Upvotes: 1

Related Questions