Using Mozmill for testing Firefox addon

Question

I have a Mozilla Firefox addon and wanna test it. I've found Mozmill and wrote the small Python script, which is just taking each Firefox version and run the command like this:

mozmill --binary=C:\browsers\firefox\38.0.1\firefox.exe 
--addon=C:\my_ext\ext 
--test=C:\my_ext\tests\mozmill\

Here is the script unit_test_runner_public.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from subprocess import check_output
import time

firefox_binary_path = 'C:\\browsers\\firefox\\'
ff_addon_location = 'C:\\my_ext\\ext'
tests_folder = 'C:\\my_ext\\tests\\mozmill\\'
ff_versions = ['38.0.1', '37.0.2', '36.0.4', '35.0.1', '34.0.5', '33.1.1',
               '32.0.3', '31.0', '30.0', '29.0.1', '28.0', '27.0.1', '26.0',
               '25.0.1', '24.0', '23.0.1', '22.0', '21.0', '20.0.1', '19.0.2',
               '18.0.2', '17.0.1', '16.0.2', '15.0.1', '14.0.1', '13.0.1']


def build_ff_path(ff_version):
    return "%s%s%s" % (firefox_binary_path, ff_version, "\\firefox.exe")


for item in ff_versions:
    print "##### Started unit tests for Mozilla Firefox %s #####" % item
    current_run = "mozmill --binary=%s --addon=%s --test=%s" % \
        (build_ff_path(item), ff_addon_location, tests_folder)
    test_run_result = check_output(current_run, shell=True)
    print "##### Finished unit tests for Mozilla Firefox %s #####" % item

    time.sleep(10)

So Mozmill is starting the browser, running the tests and then closing the browser and doing it for each Firefox version starting from 38.0.1 to 13.0.1

The problem is, that almost each time it hangs on some random Firefox version. So it opens the browser instance, run the tests, but then it's not closing the browser and the Firefox window hangs for some time and then I see such exception in the terminal:

##### Finished unit tests for Mozilla Firefox 16.0.2 #####
##### Started unit tests for Mozilla Firefox 15.0.1 #####
mozversion INFO | application_buildid: 20120905151427
mozversion INFO | application_changeset: 0b774a1067fe
mozversion INFO | application_display_name: Firefox
mozversion INFO | application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion INFO | application_name: Firefox
mozversion INFO | application_repository: http://hg.mozilla.org/releases/mozilla
-release
mozversion INFO | application_vendor: Mozilla
mozversion INFO | application_version: 15.0.1
mozversion INFO | platform_buildid: 20120905151427
mozversion INFO | platform_changeset: 0b774a1067fe
mozversion INFO | platform_repository: http://hg.mozilla.org/releases/mozilla-re
lease
mozversion INFO | platform_version: 15.0.1
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 878, in run
    mozmill.run(tests, self.options.restart)
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 473, in run
    self.stop_runner()
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 595, in stop_ru
nner
    raise Exception('client process shutdown unsuccessful')
Exception: client process shutdown unsuccessful
Traceback (most recent call last):
  File "unit_test_runner_public.py", line 24, in <module>
    test_run_result = check_output(current_run, shell=True)
  File "C:\Python27\lib\subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'mozmill --binary=C:\browsers\firefox\15.
0.1\firefox.exe --addon=C:\my_ext\ext --test=C:\my_ext\tests\mozmill\' returned 
non-zero exit status 1

C:\my_ext>

And every time it happens with the random version of Firefox, so there is no pattern like issue with some specific Firefox version.

System details are:

• OS: Microsoft Windows 7 Enterprise SP1 x86
• Python: 2.7.9
• Mozmill: 2.0.10

And the output of pip list:

blessings (1.6)
jsbridge (3.0.3)
ManifestDestiny (0.5.7)
manifestparser (1.1)
mozcrash (0.14)
mozdevice (0.45)
mozfile (1.1)
mozinfo (0.7)
mozlog (2.11)
mozmill (2.0.10)
moznetwork (0.24)
mozprocess (0.22)
mozprofile (0.23)
mozrunner (5.35)
mozversion (1.0)
pip (1.5.6)
setuptools (7.0)

Does anyone had experience with such issues?

Answer 1

So, after posting a question on Mozmill Developers Google Groups - I've got an answer from Henrik Skupin, who is responsible for Mozmill.

In short, they've also came across this issue, but Mozmill will be discontinued soon - and it's better start using the new framework firefox-ui-tests. Unfortunately it doesn't have yet possibility to specify --addon like it was with Mozmill, but bug (feature request) ;) is already created - so most likely this functionality will be added in future releases. Meanwhile, if you need to test Firefox addon, you can specify --profile to already prebuilt Firefox profile with preinstalled addon in it.