running tests that use remote webdriver and connects to a selenium hub and node fails

Question

Overview:

I am trying to run a python script that uses webdriver.Remote using CHROME browser in an ubuntu server. I have setup a selenium grid (a hub and a node) in the server and used systemd service files to run the hub and node AND xvfb as background processes. However, running the python file gives me a: selenium.common.exceptions.WebDriverException:

More Details:

Selenium server version: 3.4
Chromedriver version: 2.29 (Already in /usr/local/bin - no problem running this)
Geckodriver version: 0.15.0 (i am using chrome browser, but putting this here just in case you need it.
i am already exporting DISPLAY :5 for xvfb
checking netstat -tln shows that hub and node are running on ports 4444 and 5555
i am automating the setup of the digital ocean servers via fabric. So how do i run the python script? I put the code INSIDE the digital ocean server, and run the code there
please let me know if you need more details but when i check the logs of the NODE, i don't see any messages at all that a session has been created in the node

I've been creating fresh droplets and spent countless hours already debugging what is wrong.

Code

For xvfb process:

[Unit]
Description=X Virtual Frame Buffer Service
After=network.target

[Service]
ExecStart=/usr/bin/Xvfb :5 -screen 0 1024x768x8

[Install]
WantedBy=multi-user.target

For selenium HUB:

[Unit]
Description=Selenium Hub
After=syslog.target network.target


[Service]
User=root
Type=simple
Environment=DISPLAY=:5
ExecStart=/usr/bin/java -Dwebdriver.gecko.driver=/usr/local/bin/geckodriver -Dwebdriver.chrome.bin=/usr/bin/google-chrome -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver -jar /usr/local/bin/selenium-server-standalone.jar -role hub -log /var/log/selenium-hub/output.log


[Install] 
WantedBy=multi-user.target

And here's another one for my selenium NODE

....

[Service]
User=root
Type=simple
Environment=DISPLAY=:5
ExecStart=/usr/bin/java -Dwebdriver.gecko.driver=/usr/local/bin/geckodriver -Dwebdriver.chrome.bin=/usr/bin/google-chrome -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver -jar /usr/local/bin/selenium-server-standalone.jar -role node -hub http://localhost:4444/grid/register -log /var/log/selenium-node/output.log

Here's the error i get

selenium.common.exceptions.WebDriverException: Message: None
Stacktrace:
     at java.util.HashMap.putMapEntries (HashMap.java:500)
     at java.util.HashMap.putAll (HashMap.java:784)
     at org.openqa.selenium.remote.DesiredCapabilities. (DesiredCapabilities.java:55)
     at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:104)
....
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:672
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:590
at java.lang.Thread.run (Thread.java:745)

Here's how i use remote.WebDriver

def create_driver():
    command_executor = 'http://localhost:4444/wd/hub'  # default
    capabilities = {
        'browserName': 'chrome',
        'version': '',
        'platform': 'ANY',
        'javascriptEnabled': 'True',
        # 'acceptSslCerts': 'False',
        # 'unexpectedAlertBehaviour': 'True',
        'chromeOptions': {
            'args': [
                '--disable-extensions',
                '--no-sandbox',
                '--disable-setuid-sandbox',
                '--verbose',
                '--log-path=/var/log/chromedriver-full.log',
                '--disable-impl-side-painting',
                '--memory-model=high',
                '--disk-cache-size=500000000',
                '--allow-running-insecure-content',
                '--ignore-certificate-errors',
                '--ignore-urlfetcher-cert-requests',
                '--disable-gpu',
                '--disk-cache-dir=null',
            ],
            # 'extensions': [],
            # 'binary': '/usr/bin/chromium-browser',
            # 'minidumpPath': '/root/'
        },
        'loggingPrefs': {'driver': 'ALL', 'server': 'ALL', 'browser': 'ALL', 'performance': 'ALL'}
    }
    driver = webdriver.Remote(desired_capabilities=capabilities, command_executor=command_executor)
    time.sleep(5)  # Let the user actually see something!
    driver.implicitly_wait(60)
    return driver


def main():
    print("Running remote web driver test")
    driver = create_driver()
    driver.get("https://www.google.com.ph")
    print(driver.current_url)
    screenshot = driver.save_screenshot('/root/test.png')
    print(screenshot)
    driver.quit()
    display.stop()
    print("Success test on remote web driver")

main()