org.openqa.selenium.WebDriverException: connection refused while using Selenium Java and webdriver (linux headless with Xvfb)

Question

Recently i got a new virtual server which im trying to configure since some days to make my Java (8) based selenium web app running. This webapp is running inside a tomcat (8.5.31) and is using selenium for some basic website scraping. The OS of the server is gentoo (before i had a debian system which was running fine)

I spend already some days to make it work but now im stuck at the following error:

1526203737341   geckodriver     INFO    geckodriver 0.20.1
1526203737367   geckodriver     INFO    Listening on 127.0.0.1:25016
1526203737796   mozrunner::runner       INFO    Running command: "/usr/bin/firefox-bin" "-marionette" "-profile" "/tmp/rust_mozprofile.ESuH8rdqpwT8"
1526203737802   geckodriver::marionette DEBUG   Waiting 60s to connect to browser
1526203739872   Marionette      INFO    Listening on port 2828
1526203797867   webdriver::server       DEBUG   <- 500 Internal Server Error {"value":{"error":"unknown error","message":"connection refused","stacktrace":""}}
11:29:57.944 [pool-2-thread-1] ERROR test.Test - Exception occured while processing.
org.openqa.selenium.WebDriverException: connection refused
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'gentoo', ip: 'xxx.xxx.xxx.xxx', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.95-gentoo', java.version: '1.8.0_172'
Driver info: driver.version: FirefoxDriver

This is what the process list is showing:

root      7808     1  0 14:36 ?        00:00:00   /usr/bin/Xvfb :99 -screen 0 1600x1200x24 -auth /etc/conf.d/Xvfb.cfg -listen tcp
tomcat    7863     1 59 14:38 ?        00:00:51   /etc/java-config-2/current-system-vm/bin/java -Djava.util.logging.config.file=/opt/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderL
tomcat    7923  7863  0 14:38 ?        00:00:00     /opt/geckodriver --port=31399 -b /usr/bin/firefox-bin
tomcat    7943  7923  6 14:38 ?        00:00:04       /opt/firefox/firefox -marionette --headless -profile /tmp/rust_mozprofile.qpQfCj8DR1ab
tomcat    8032  7943  0 14:38 ?        00:00:00         /opt/firefox/plugin-container -greomni /opt/firefox/omni.ja -appomni /opt/firefox/browser/omni.ja -appdir /opt/firefox/browser 7943 true tab
tomcat    7977     1  0 14:38 ?        00:00:00   dbus-launch --autolaunch 32b951aa6b930bec1f195b765af588f5 --binary-syntax --close-stderr
tomcat    7978     1  0 14:38 ?        00:00:00   /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
tomcat    7993     1  0 14:38 ?        00:00:00   /usr/libexec/gconfd-2

The Code for creating the Firefox instance (was using more detailed log levels to get some more informations about the problem):

    System.setProperty("webdriver.gecko.driver", driverGeckoPath);

    LoggingPreferences loggingPrefs = new LoggingPreferences();
    loggingPrefs.enable(LogType.BROWSER, Level.ALL);
    loggingPrefs.enable(LogType.CLIENT, Level.ALL);
    loggingPrefs.enable(LogType.DRIVER, Level.ALL);
    loggingPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    loggingPrefs.enable(LogType.PROFILER, Level.ALL);
    loggingPrefs.enable(LogType.SERVER, Level.ALL);

    DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
    desiredCapabilities.setCapability("marionette", true);
    desiredCapabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
    desiredCapabilities.setCapability(CapabilityType.LOGGING_PREFS, loggingPrefs);

    FirefoxOptions options = new FirefoxOptions(desiredCapabilities);
    options.setLogLevel(FirefoxDriverLogLevel.TRACE);

    driver = new FirefoxDriver(options);

The geckodriver lays in /opt/ and is executable.

My current setup/configuration

Selenium
All components are having version 3.12.0 (selenium-java, selenium-remote-driver, selenium-firefox-driver, selenium-api)

Firefox
Version 60.0

geckodriver
Version 20.1

Xvfb configuration

XVFB=/usr/bin/Xvfb
XVFBARGS=":99 -screen 0 1600x1200x24 -auth /etc/conf.d/Xvfb.cfg -nolisten tcp"
PIDFILE=/var/run/Xvfb.pid

Xvfb config

localhost

Xvfb init.d-script

start() {
        ebegin "Starting virtual X frame buffer: Xvfb"
        start-stop-daemon --start --pidfile $PIDFILE --make-pidfile --background --exec $XVFB -- $XVFBARGS
        eend ${?}
}

stop() {
        ebegin  "Stopping virtual X frame buffer: Xvfb"
        start-stop-daemon --stop --pidfile $PIDFILE
        eend ${?}
}

Export of display variable (initialized at boot time for all users):

DISPLAY=:99

/etc/conf.d/hostname

gentoo

/etc/hosts

127.0.0.1       gentoo
127.0.0.1       localhost
::1             localhost

Would appreciate very much any hints/tips to get more informations about how to debug/solve this problem...

Update 1

I tried running my selenium component with the ChromeDriver which interestingly works. The code:

  System.setProperty("webdriver.chrome.driver", "/opt/chromedriver");
  driver = new ChromeDriver();

The log shows:

Starting ChromeDriver 2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb) on port 24646
Only local connections are allowed.

Update 2

When im running the app in my local environment (similary configured / gentoo) everything works fine. But im not using Xvfb here but a xorg-server instead. This is the logging output:

1526217622830   geckodriver INFO    geckodriver 0.20.1
1526217622833   geckodriver INFO    Listening on 127.0.0.1:12500
1526217622955   mozrunner::runner   INFO    Running command: "/usr/bin/firefox-bin" "-marionette" "-profile" "/tmp/rust_mozprofile.AZMw0iETRqFj"
1526217622955   geckodriver::marionette DEBUG   Waiting 60s to connect to browser
1526217623246   Marionette  DEBUG   Received observer notification profile-after-change
1526217623372   Marionette  DEBUG   Received observer notification command-line-startup
1526217623372   Marionette  DEBUG   Received observer notification nsPref:changed
1526217624744   Marionette  DEBUG   Received observer notification sessionstore-windows-restored
1526217624816   Marionette  DEBUG   Setting recommended pref toolkit.cosmeticAnimations.enabled to false
1526217624816   Marionette  DEBUG   Setting recommended pref datareporting.policy.dataSubmissionPolicyAccepted to false
1526217624816   Marionette  DEBUG   Setting recommended pref extensions.shield-recipe-client.api_url to 
1526217624847   Marionette  DEBUG   New connections are accepted
1526217624848   Marionette  INFO    Listening on port 34253
1526217624860   geckodriver::marionette DEBUG   Connected to Marionette on localhost:34253
1526217624868   Marionette  DEBUG   Accepted connection 0 from 127.0.0.1:56070

For me it seems to be a connection problem between Marionette and the geckodriver. Could it be a port problem? On the server Marionette seems to listen always at port 2828, on my local machine its seems to be always a random port...

Answer 1

For those whom it may concern I had the very same problem and got stuck in "connection refused" fiesta. After two whole night of try and error I found out that my previous processes of geckodriver, selenium and firefox was not closed and I had something around 50 open process. This three simple commands solved my problem

pkill geckodriver
pkill firefox
pkill selenium

Answer 2

This error message...

1526203737802   geckodriver::marionette DEBUG   Waiting 60s to connect to browser
1526203739872   Marionette      INFO    Listening on port 2828
1526203797867   webdriver::server       DEBUG   <- 500 Internal Server Error {"value":{"error":"unknown error","message":"connection refused","stacktrace":""}}
11:29:57.944 [pool-2-thread-1] ERROR test.test.test.test.test.Test - Exception occured while processing.
org.openqa.selenium.WebDriverException: connection refused

...implies that the GeckoDriver was unable to initiate/spawn a new Web Client session/instance successfully.

Your main issue is the way you have assigned the DesiredCapabilities type of object i.e. desiredCapabilities to the FirefoxOptions i.e to options. The proper way would be to pass the DesiredCapabilities class object i.e. desiredCapabilities through merge() function from MutableCapabilities class into the FirefoxOptions class object as follows : :

System.setProperty("webdriver.gecko.driver", driverGeckoPath);

LoggingPreferences loggingPrefs = new LoggingPreferences();
loggingPrefs.enable(LogType.BROWSER, Level.ALL);
loggingPrefs.enable(LogType.CLIENT, Level.ALL);
loggingPrefs.enable(LogType.DRIVER, Level.ALL);
loggingPrefs.enable(LogType.PERFORMANCE, Level.ALL);
loggingPrefs.enable(LogType.PROFILER, Level.ALL);
loggingPrefs.enable(LogType.SERVER, Level.ALL);

DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.setCapability("marionette", true);
desiredCapabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
desiredCapabilities.setCapability(CapabilityType.LOGGING_PREFS, loggingPrefs);

FirefoxOptions options = new FirefoxOptions();
options.merge(desiredCapabilities);
options.setLogLevel(FirefoxDriverLogLevel.TRACE);

driver = new FirefoxDriver(options);

---

Update

As you are still facing the same error consider performing the following steps :

• Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
• Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
• If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
• Take a System Reboot.
• Execute your Test as a non-root user.
• Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.