How To Configure Chrome Browser To Run In Headless Mode

Question

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

from time import sleep

# Gumtree credentials
username = "my username"
password = "my password"

# Removes SSL Issues With Chrome
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('log-level=3') 
options.add_argument('--disable-notifications')

# Initiate Headless Chrome
#options.add_argument('--headless')
#options.add_argument('--disable-gpu')
#options.headless = True

# Initiate Chrome Driver
url = 'https://my.gumtree.com/login'
driver = webdriver.Chrome(executable_path="C:\webdrivers\chromedriver.exe",options=options)
driver.get(url)

# Find Username/Email Field and Send to Input Field
driver.find_element_by_id("email").send_keys(username)
# Find Password Field and Send to Input Field
driver.find_element_by_id("fld-password").send_keys(password)

# Initiate Consent Button
consent_button_xpath = '//*[@id="login-form"]/div/form/fieldset/div[3]/button'
consent = WebDriverWait(driver, 40).until(EC.element_to_be_clickable((By.XPATH, consent_button_xpath)))
consent = driver.find_element_by_xpath(consent_button_xpath)
consent.click()

# Print Username to Test Successful Login
login_username = driver.find_element_by_xpath('//*[@id="is-being-refined"]/div[3]/div/header/div[1]/div/nav/div/ul/li[5]/a/div').text
print(login_username)

# close the driver
driver.close()

The above script successfully automates the login process into a website using Selenium and prints the username.

Issue

I'm trying to carry out the above in silent mode using Headless Chrome without any joy.
I have researched many articles and simply adding the switch options.add_argument('--headless') causes the script to fail.

Any help would be much appreciated.

Error Message

[0830/063818.313:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -101
[0830/063818.313:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -101
Traceback (most recent call last):
  File "e:\Python Projects\Gumtree\test5.py", line 39, in <module>
    consent.click()
  File "C:\Python39\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "C:\Python39\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button class="btn-primary btn-full-width" type="submit" data-analytics="gaEvent:...Attempt,userData:{lip:Email}">Login</button> is not clickable at point (386, 564). Other element would receive the click: <button id="onetrust-pc-btn-handler" tabindex="0" class="cookie-setting-link">...</button>
  (Session info: headless chrome=92.0.4515.159)

Answer 1

This error :

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element Login is not clickable at point (386, 564). Other element would receive the click: ... (Session info: headless chrome=92.0.4515.159)

does not have anything to do with selenium not switching to headless mode.

The moment you write

options.add_argument("--headless")

and pass this option reference in webdriver object creation call, Selenium will treat this as a headless mode.

1. Also I would suggest to use explicit waits/Implicit waits, whenever you think it's required.

2. Headless has never been stable, causing issues here and there. If you just wanna do web scraping then consider different tool like BS4 or request module.

3. Also you should use relative xpath rather than absolute xpath.

Answer 2

I'd recommend not using mouse click for the submit button but use the keyboard instead:

from selenium.webdriver.common.keys import Keys
consent.send_keys(Keys.ENTER)