Chromedriver remote-debugging-port with Selenium

Question

I am using Capybara Selenium to run headless Chrome, which works great, except I cannot figure out how to use remote debugging. When I add --remote-debugging-port=4444 or --remote-debugging-port=9222 or --remote-debugging-port=9521, Selenium no longer connects to the browser to run the test.

How do I get remote debugging to work? Here is my code for reference:

Capybara.register_driver :selenium do |app|
  # from https://github.com/SeleniumHQ/selenium/issues/3738
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(loggingPrefs: {browser: 'ALL'})
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument '--disable-infobars' # hide info bar about chrome automating test
  # if we don't use this flag, every selenium test will die with the error:
  # "unknown error: Chrome failed to start: exited abnormally"
  options.add_argument '--no-sandbox'
  # BREAKS THINGS if uncommented
  # options.add_argument '--remote-debugging-port=4444'
  options.add_argument '--headless'
  options.add_argument '--window-size=1600,2400'
  options.add_preference('profile.default_content_settings.popups', 0)
  options.add_preference('download.default_directory', DownloadHelpers::PATH.to_s)
  Capybara::Selenium::Driver.new(
    app,
    clear_local_storage: true,
    clear_session_storage: true,
    browser: :chrome,
    options: options,
    desired_capabilities: capabilities,
  )
end

Answer 1

---

Update: if using versions since Chrome 67/chromedriver 2.39, my alternative answer above provides a simpler solution

---

The core issue here is Chromedriver also uses the remote debugging port connection to communicate with Chrome. This uses the websocket protocol, which only supports a single-client connected at a time. Normally, when chromedriver starts the chromedriver process, it will choose a random free TCP port number and use this to access the remote debug port. If you specify --remote-debuggging-port=9222 however, Chrome will be opened with the debug port you have requested, but chromedriver will silently continue to try and open a connection using this random port number.

The solution I ended up with was heavily inspired by comment 20 in this chromedriver issue. It required quite a bit of code to get it working, but works solidly. It makes uses of a nodejs library, crmux - which allows multiple clients to connect to the remote debug port of chrome at the same time.

1. Get nodejs installed first: Nodejs v9.7.0 works fine
2. Install crmux by running npm install crmux -g
3. Before you start chromedriver (Capybara::Selenium::Driver.new), you need to spawn a separate thread that will do a few things: look for the remote debug port chromedriver is trying to use to connect to chrome, and then use this to fire up crmux. Once this has happened Capybara etc will work as normal.
4. My separate thread runs a ruby script that first executes a netstat command repeatedly until it finds the relevant entry for chromedriver (TCP status is SYN_SENT). This separate thread must continue to run in the background while chrome is up and running.

The code for this is:

$chrdrv_wait_timeout = 60
$chrdrv_exe = "chromedriver.exe"

def get_netstat_output
  stdout = `netstat -a -b -n`
  stat_lines = stdout.split("\n")
  stat_lines
end

def try_get_requested_port
  socket_state = "SYN_SENT" # i.e. sent with no reply
  statout = get_netstat_output
  n = statout.length
  i = 0
  loop do
    i += 1
    # find lines relating to chromedriver
    exe_match = /^ +\[#{$chrdrv_exe}\]$/.match statout[i]
    if exe_match != nil
      # check preceeding lines which should contain port info
      port_match = /TCP.*:([0-9]+)\W+#{socket_state}/.match statout[i-1]
      if port_match != nil
        return port_match[1].to_i
      end
    end
    break unless i < n
  end
  return nil
end

def get_tcp_port_requested_by_chromedriver
  i = 1
  loop do
    puts "Waiting for #{$chrdrv_exe}: #{i}"
    port = try_get_requested_port
    if port != nil
      return port
    end
    break unless i < $chrdrv_wait_timeout
    sleep 1
    i += 1
  end
  raise Exception, "Failed to get TCP port requested by #{$chrdrv_exe} (gave up after #{$chrdrv_wait_timeout} seconds)"
end

(I'm working in Windows: for Mac/Linux the netstat syntax/output is probably different so the code will need adjustment; the key thing is you need it to output the executable owner of each connection entry - and parse the bit relating to chromedriver to get the port in question).

5. Once this has found the random port (I'll use 12225 as an example), the background ruby script can then execute a crmux process, which will reunite chromedriver with chrome itself via the port you specified in Capybara (4444):

   crmux --port=4444 --listen=12225

6. Finally, this separate script saves the discovered listen port to a text file. This allows the main script/thread that is running capybara to know the port number it can use to get access to chrome (via crmux's multiplexed connection) by reading in the port from that file. So you can then use chrome_remote to access chrome using port 12225, for example, while capybara is doing its thing.

Answer 2

Since chrome 67 and chromedriver 2.39, chromedriver now correctly uses the port you specify with --remote-debugging-port. This removes quite a bit of complexity from my answer above. The steps I now take, which work for my use case of needing to configure download settings using chrome_remote, are as follows:

It makes uses of a nodejs library, crmux - which allows multiple clients to connect to the remote debug port of chrome at the same time.

1. Get nodejs installed first: Nodejs v9.7.0 works fine
2. Install crmux by running npm install crmux -g

3. Before you start chromedriver (Capybara::Selenium::Driver.new), you need to spawn a separate thread that will fire up crmux, which will let both you and chromedriver communicate with chrome itself via the port you specified in Capybara (4444):

   crmux --port=4444 --listen=4444

4. You may want to add a sleep 3 after the spawn command in the main script/thread to give time for crmux to start before you continue with your test startup.

You can then use chrome_remote (for example) to access chrome using port 4444, while capybara is doing its thing.

Answer 3

Updating my ChromeDriver fixed it for me. I didn't have to do anything else. Before it would hang when attempting to start the test.

Specifically I was on ChromeDriver 2.36 and I upgraded to ChromeDriver 2.40. I don't think the Chrome version was a problem, since I was on Chrome 67 both before and after.

Here's how I'm registering the driver:

Capybara.register_driver :headless_chrome do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    chromeOptions: { args: %w[headless window-size=1280,960 remote-debugging-port=9222] }
  )
  Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
end

After that I ran my test with a debugger (binding.pry) placed where I wanted to inspect. Then when I hit the debugger I navigated to http://localhost:9222/ in a normal instance of Chrome and was able to follow a link to view what was happening in the headless Chrome instance, including the browser console output I needed.