What is the latest way to set up Headless Selenium

Question

Sorry but I'm quite new to Python. I have checked online and found that there are many videos with ways to create headless google selenium. However, upon trying those methods, I realized that they are all deprecated. Can anyone update me with the latest method to enable headless Selenium for google Chrome?

So the sequence of the program follows:

1. Enters the login page website "https://info.bbdc.sg/members-login/"
2. Then, it goes to submit insecure page "http://www.bbdc.sg/bbdc/bbdc_web/header2.asp" photo
3. Lastly, it would navigate to the home page of the website which is "http://www.bbdc.sg/bbdc/b-mainframe.asp"

[Updated] I have tried adding the suggested answer to my code. This is my new code:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options

browser_options = Options()
browser_options.add_argument("--allow-running-insecure-content")
browser_options.headless = True
browser = webdriver.Chrome(executable_path=r'C:\Users\Jonathan\Projects\drivers\chromedriver_win32_V96\chromedriver.exe',options=browser_options)
browser.get("https://info.bbdc.sg/members-login/")

login_refresh = True
while login_refresh:
    try:
        if EC.url_matches('https://info.bbdc.sg/members-login/?err=Please+refresh+your+browser+and+fill+in+the+fields'
                          '+properly+to+login.+' or 'https://info.bbdc.sg/members-login/'):
            browser.refresh()
            login_id_elem = browser.find_element(By.ID, 'txtNRIC')
            login_pw_elem = browser.find_element(By.ID, 'txtPassword')
            access_btn = browser.find_element(By.ID, 'loginbtn')
            login_id_elem.send_keys("094H28022001")
            login_pw_elem.send_keys("258046")
            time.sleep(5)
            access_btn.click()
    except NoSuchElementException:
        login_refresh = False

print(browser.current_url)
WebDriverWait(browser, 10).until(EC.url_matches('http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'))
send_anyway = browser.find_element(By.ID, 'proceed-button')
print(browser.current_url)
send_anyway.click()

# Home page
print(browser.current_url)
frame_element = browser.find_element(By.XPATH, '//frame[@name="leftFrame"]')
browser.switch_to.frame(frame_element)

However, I get this stacktrace error while running headless chrome. If i were to use the normal headful chrome, it works perfectly fine.

DevTools listening on ws://127.0.0.1:57679/devtools/browser/34af1be6-ebe7-4326-a4b5-072d940d9684
[1213/220648.128:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220648.168:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.168:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.315:INFO:CONSOLE(825)] "Uncaught TypeError: Cannot read properties of null (reading 'addEventListener')", source: https:/
/info.bbdc.sg/members-login/ (825)
[1213/220648.334:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.browser is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-j
query-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.334:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.354:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.attrFn is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-jq
uery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.355:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.370:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.unload() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enab
le-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.370:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.013:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220649.030:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.030:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.040:INFO:CONSOLE(825)] "Uncaught TypeError: Cannot read properties of null (reading 'addEventListener')", source: https:/
/info.bbdc.sg/members-login/ (825)
[1213/220649.044:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.browser is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-j
query-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.044:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.062:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.attrFn is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-jq
uery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.063:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.086:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.unload() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enab
le-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.087:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220657.962:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220657.977:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220657.977:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220657.979:INFO:CONSOLE(385)] "Mixed Content: The page at 'https://info.bbdc.sg/members-login-redirect/' was loaded over a secur
e connection, but contains a form that targets an insecure endpoint 'http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'. This endpoint shou
ld be made available over a secure connection.", source: https://info.bbdc.sg/members-login-redirect/ (385)
[1213/220657.993:INFO:CONSOLE(388)] "Mixed Content: The page at 'https://info.bbdc.sg/members-login-redirect/' was loaded over a secur
e connection, but contains a form that targets an insecure endpoint 'http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'. This endpoint shou
ld be made available over a secure connection.", source: https://info.bbdc.sg/members-login-redirect/ (388)
http://www.bbdc.sg/bbdc/b-mainframe.asp
Traceback (most recent call last):
  File "C:\Users\Jonathan\PycharmProjects\BBDCBOT\BBDC_trial_headless.py", line 33, in <module>
    WebDriverWait(browser, 10).until(EC.url_matches('http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'))
  File "C:\Users\Jonathan\venvs\automation\lib\site-packages\selenium\webdriver\support\wait.py", line 89, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Answer 1

I know two ways, and they works fine.

1. Using add_argument headless

   options = webdriver.ChromeOptions()
   options.add_argument("--headless")
   driver = webdriver.Chrome(options = options)
   

2. Using .headless = True

   options = webdriver.ChromeOptions()
   options.headless = True
   driver = webdriver.Chrome(options = options)
   

Answer 2

For selenium 4 desired capabilities replaced with browser options.

from selenium.webdriver.chrome.options import Options
 
browser_options = Options()
browser_options.headless = True
driver = webdriver.Chrome(executable_path='G:\chromedriver\chromedriver.exe',options=browser_options)