How do I test Chrome Extension/Firefox WebExtension code?

Question

Since Firefox is forcing me to, I'm rewriting my extension to use the WebExtension APIs, i.e. Chrome's Extension APIs. I want to have automated testing. So far I've tried this:

I've got a package.json so that npm will install depedencies:

{
  "name": "extension-api-tests",
  "version": "0.0.1",
  "scripts": {
    "test": "karma start"
  },
  "devDependencies": {
    "karma": "^1.3.0",
    "karma-firefox-launcher": "^1.0.0",
    "karma-mocha": "^1.3.0",
    "karma-sinon-chrome": "^0.2.0",
    "mocha": "^3.1.2",
    "sinon-chrome": "^2.1.2"
  }
}

I've got a karma.conf.js to set up that test runner:

module.exports = function(config) {
  config.set({
    frameworks: ['mocha', 'sinon-chrome'],
    files: ['test.js'],
    reporters: ['dots'],
    autoWatch: false,
    browsers: ['Firefox'],
    singleRun: true,
    concurrency: Infinity,
  });
};

And I've got basic tests:

describe('my frustration', () => {
  it('works when it uses no APIs', done => {
    done();
  });

  it('should respond to messages!', done => {
    console.log('message test starting');
    chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
      console.log('received message');
      sendResponse(true);
    });
    chrome.runtime.sendMessage({}, result => {
      console.log('received response to message');
      done();
    });
  });

  it('should open a tab!', done => {
    console.log('tab test starting');
    chrome.tabs.create({
      'active': true,
      'url': 'http://www.example.com/',
    }, tab => {
      console.log('created a tab:', tab);
      done();
    });
  });
});

This is of course a reduced test case. When I run npm test, I get (abbreviated slightly):

> extension-api-tests@0.0.1 test .../ext-test
> karma start

25 07 2017 11:57:10.395:INFO [karma]: Karma v1.7.0 server started at http://0.0.0.0:9876/
25 07 2017 11:57:10.397:INFO [launcher]: Launching browser Firefox with unlimited concurrency
25 07 2017 11:57:10.404:INFO [launcher]: Starting browser Firefox
25 07 2017 11:57:14.687:INFO [Firefox 54.0.0 (Ubuntu 0.0.0)]: Connected on socket iIjNRRQfzWj68_GNAAAA with id 42440302
.
LOG: 'message test starting'
Firefox 54.0.0 (Ubuntu 0.0.0) my frustration should respond to messages! FAILED
        Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
LOG: 'tab test starting'
Firefox 54.0.0 (Ubuntu 0.0.0) my frustration should open a tab! FAILED
        Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
Firefox 54.0.0 (Ubuntu 0.0.0): Executed 3 of 3 (2 FAILED) (3.998 secs / 4.001 secs)
npm ERR! Test failed.  See above for more details.

My tests which try to use extension APIs all fail. It does not say (e.g.) chrome.runtime is not defined (but it does if I remove 'sinon-chrome' from karma.conf.js), so I believe I have sinon set up. But the APIs never do anything, never work. The code I want to test is all about passing data around through these APIs (especially as messages, to cross the chrome/content boundary).

Answer 1

In addition to all the setup in the original question, realize that Sinon provides only stubs/mocks of the API surface: not a fake API implementation.

For the test to "work", one must declare the behavior of the mock as well; see e.g. https://sinonjs.org/releases/latest/mocks/ and https://sinonjs.org/releases/latest/stubs/ . The sort of test in the original question is "not possible", as you'd primarily be testing the fake implementation which you'd also need to write in the test.

Once set up sinon-chrome gives you the ability to write tests more like

chrome.tabs.executeScript.callsArg(2);
// ... call code under test ...    
assert(chrome.tabs.executeScript.calledOnce);

(Which are not the sort of tests I really want to be writing.)

Answer 2

According to https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/runtime/sendMessage sendMessage returns a Promise. According to your logs, your success handler in

chrome.runtime.sendMessage({}, result => {
  console.log('received response to message');
  done();
});

is not being called in time compared to the main thread that runs the test. You should try to add a sleep to the main thread

function sleep (time) {
  return new Promise((resolve) => setTimeout(resolve, time));
}

like so

chrome.runtime.sendMessage({}, result => {
  console.log('received response to message');
});

sleep(500).then(() => {
  done();
});

As long as sendMessage returns within 500 ms, this should work.