running python selenium with firefox on headless raspberry pi

Question

I am trying to get selenium working on my headless raspberry pi with firefox. I have it working fine on Windows with chrome. Here are my versions:

uname -a Linux megabyte.thompco.com 4.9.59-v7+ #1047 SMP Sun Oct 29
12:19:23 GMT 2017 armv7l GNU/Linux

which firefox
/usr/bin/firefox

firefox --version
Mozilla Firefox 52.5.2

./geckodriver_32 --version
geckodriver 0.19.1

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

I think I have compatible versions of the driver and firefox (this seems to work):

./geckodriver_32 -b /usr/bin/firefox
1516245181824   geckodriver     INFO    geckodriver 0.19.1
1516245181881   geckodriver     INFO    Listening on 127.0.0.1:4444

When I run the following code:

def __init__(self, tag, user_name, password, driver_location, headless):
    logger = logging_utils.get_logger()
    logging_utils.start_function(logger, user_name=user_name)
    self.tag = tag
    self.user_name = user_name
    self.password = password
    self.cards = []
    driver_options = Options()
    driver = None
    try:
        if "chrome" in driver_location.lower():
            if headless:
                driver_options.add_argument("--headless")
            driver = webdriver.Chrome(executable_path=os.path.abspath("chromedriver.exe"),
                                      chrome_options=driver_options)
        elif "gecko" in driver_location.lower():
            binary = FirefoxBinary("/usr/bin/firefox")
            driver_options.binary = binary
            profile = webdriver.FirefoxProfile()
            driver_options.profile = profile
            driver_options.set_headless(headless)
            driver = webdriver.Firefox(firefox_binary=binary,
                                       firefox_profile=profile,
                                       executable_path=os.path.abspath(driver_location),
                                       firefox_options=driver_options)

I get this error:

Traceback (most recent call last):

  File "/mnt/usbdrive/python/AmexOfferChecker/amexParser.py", line 105, in __init__
    firefox_options=driver_options)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
    keep_alive=True)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
    self.start_session(desired_capabilities, browser_profile)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
    raise exception_class(message, screen, stacktrace)

SessionNotCreatedException: Message: Unable to find a matching set of capabilities

Any suggestions would be most welcome!

I have modified my "gecko" section to look like this:

            options = Options()
            options.add_argument('-headless')
            print driver_location
            print os.path.abspath(driver_location)
            driver = Firefox(executable_path=os.path.abspath(driver_location),
                             firefox_options=options)
            print "Driver has been loaded!"

Now I get this error:

  geckodriver_32
  /mnt/usbdrive/python/AmexOfferChecker/geckodriver_32
  Traceback (most recent call last):

  File "/mnt/usbdrive/python/AmexOfferChecker/amexParser.py", line 106, in __init__
    firefox_options=options)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
    keep_alive=True)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
    self.start_session(desired_capabilities, browser_profile)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)

  File "/home/jordan/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
    raise exception_class(message, screen, stacktrace)

WebDriverException: Message: Process unexpectedly closed with status: 1

I did see this bug: On Linux, certain libraries are currently required on your system even though headless mode doesn't use them — because Firefox links against them. See bug 1372998 for more details and progress towards a fix. I added the suggested libraries:

sudo apt-get install libgtk-3-0 libdbus-glib-1-2 xvfb

but am still getting the same exception.

I have tried this also with the same error (NOTE that the debug file is created, but empty):

            options = Options()
            options.add_argument('--headless')
            profile = webdriver.FirefoxProfile()   
            profile.set_preference("logs/webdriver.log", "/tmp/firefox_console")
            binary = FirefoxBinary(firefox_path="/usr/bin/firefox",
                                   log_file=open("/tmp/firefox_output", "wb"))
            driver = webdriver.Firefox(firefox_profile=profile,
                                       firefox_binary=binary,
                                       options=options,
                                       executable_path=os.path.abspath(driver_location))

Finally got logging turned on for seleniuim (does this mean anything to anyone?):

2018-01-24 22:51:00,078 - selenium.webdriver.remote.remote_connection 480 -DEBUG - POST http://127.0.0.1:45413/session {"capabilities": {"alwaysMatch": {"acceptInsecureCerts": true, "browserName": "firefox", "moz:firefoxOptions": {"args": ["headless"]}}, "firstMatch": [{}]}, "desiredCapabilities": {"acceptInsecureCerts": true, "browserName": "firefox", "moz:firefoxOptions": {"args": ["headless"]}}}

2018-01-24 22:51:00,944 - selenium.webdriver.remote.remote_connection 567 -DEBUG - Finished Request {"value":{"error":"unknown error","message":"Process unexpectedly closed with status: 1","stacktrace":"stack backtrace:\n   0:   0x55d797 - backtrace::backtrace::trace::hc4bd56a2f176de7e\n   1:   0x55d8ff - backtrace::capture::Backtrace::new::he3b2a15d39027c46\n   2:   0x4b7f4b - webdriver::error::WebDriverError::new::ha0fbd6d1a1131b43\n   3:   0x4bcb57 - geckodriver::marionette::MarionetteHandler::create_connection::hf0532ddb9e159684\n   4:   0x4a14cb - <webdriver::server::Dispatcher<T, U>>::run::h2119c674d7b88193\n   5:   0x47fcbf - std::sys_common::backtrace::__rust_begin_short_backtrace::h21d98a9ff86d4c25\n   6:   0x4871cf - std::panicking::try::do_call::h5cff0c9b18cfdbba\n   7:   0x606237 - panic_unwind::__rust_maybe_catch_panic\n                at /checkout/src/libpanic_unwind/lib.rs:99\n   8:   0x4999e7 - <F as alloc::boxed::FnBox<A>>::call_box::h413eb1d9d9f1c473\n   9:   0x6000d3 - alloc::boxed::{{impl}}::call_once<(),()>\n                at /checkout/src/liballoc/boxed.rs:692\n                 - std::sys_common::thread::start_thread\n                at /checkout/src/libstd/sys_common/thread.rs:21\n                 - std::sys::imp::thread::{{impl}}::new::thread_start\n                at /checkout/src/libstd/sys/unix/thread.rs:84"}}
2018-01-24 22:51:00,947 - main.main 38 -WARNING - Problem (Message: Process unexpectedly closed with status: 1

Here is the crux of the error (interesting that it is reported as a DEBUG). does anyone have any suggestions:

2018-01-24 22:51:02,863 - selenium.webdriver.remote.remote_connection 567 -DEBUG - Finished Request 
{"value":
    {"error":"unknown error","message":"Process unexpectedly closed with status: 1","stacktrace":"stack backtrace:
    0:   0x576797 - backtrace::backtrace::trace::hc4bd56a2f176de7e
    1:   0x5768ff - backtrace::capture::Backtrace::new::he3b2a15d39027c46
    2:   0x4d0f4b - webdriver::error::WebDriverError::new::ha0fbd6d1a1131b43
    3:   0x4d5b57 - geckodriver::marionette::MarionetteHandler::create_connection::hf0532ddb9e159684
    4:   0x4ba4cb - <webdriver::server::Dispatcher<T, U>>::run::h2119c674d7b88193
    5:   0x498cbf - std::sys_common::backtrace::__rust_begin_short_backtrace::h21d98a9ff86d4c25
    6:   0x4a01cf - std::panicking::try::do_call::h5cff0c9b18cfdbba
    7:   0x61f237 - panic_unwind::__rust_maybe_catch_panic
    at /checkout/src/libpanic_unwind/lib.rs:99
    8:   0x4b29e7 - <F as alloc::boxed::FnBox<A>>::call_box::h413eb1d9d9f1c473
    9:   0x6190d3 - alloc::boxed::{{impl}}::call_once<(),()>
    at /checkout/src/liballoc/boxed.rs:692
    - std::sys_common::thread::start_thread
    at /checkout/src/libstd/sys_common/thread.rs:21
    - std::sys::imp::thread::{{impl}}::new::thread_start
    at /checkout/src/libstd/sys/unix/thread.rs:84"
    }
}

Answer 1

You can also try declaring the DISPLAY variable, it works especially for remote connections. Run this command on the terminal:

export DISPLAY=:0.0

Answer 2

Finally got this to work, but with chromedriver and chrome. You will have to install chrome first:

sudo apt-get install chromium-browser

Next downloaded the debian package from here: https://packages.debian.org/stretch/armhf/chromium-driver/download Unpack the file "chromedriver":

mkdir tmp
dpkg-deb -R chromium-driver_63.0.3239.84-1_deb9u1_armhf.deb tmp
cp /usr/local/bin/chromedriver .
mv chromedriver chromedriver_arm_64

The rest of the code is unchanged. Note that

        driver_options.add_argument("headless")

is fine ("--headless" may work as well - I haven't tried it). I sure hope that someone finds this before spending as much time on this as I have!