Cannot use Private Socks5 proxy with Selenium+Firefox (Python)

Question

Context: I am using Python, selenium + Geckodriver. Firefox webbrowser automation and I have problem on a private proxy authentification. I cannot manage to configure my socks5 proxy username and password. Thus, Firefox shows up but connection doesnt work.

The simple task that I want to do is :

1. I want to use selenium(automated Firefox) with a proxy (SOCKS5 proxy with password)
2. Checking my IP on a website to see if the proxy is working

What I noted so far is:

• I am using Firefox V76.0.1 , Geckodriver and Python3
• The User / Password box that should normally pop up when u choose doesn't pop up, this could be a problem with regards to the Firefox version and it could be fixed by overriding some settings in : about:setings. A solution could be to make this popup work and write some User password auth - see login function below. Honestly , it is unstable I prefer to find a solution that can work without this popup.
• The only stable way to manage socks5 proxies (regardless of the popup authentification) is using an AddOn called FoxyProxy.
• I cannot change this (Foxy Proxy AddOn) and all I can change in python with regards to Firefox are the settings that you can acess in Firefox using the URL (about:config)
• Simple non password protected proxies are not a problem but SOCKS5 proxies with username and pasword are a problem
• Also: wondering if it is possible to override Foxy Proxy AddOn and add options that can be accessible in Firefox (about:config) ( Honestly I am not a Firefox plugin developper so maybe it is possible!? )

So the current code is, replace HOST/USERNAME/PORT/PASSWORD with socks5 proxy data :

    from selenium import webdriver
    from base64 import b64encode
    from selenium import webdriver

    def login(browser):
        alert=browser.switch_to_alert()
        alert.send_keys("username"+webdriver.common.keys.Keys.TAB+"password")
        alert.accept()

    proxy = {'host': HOST, 'port': PORT, 'usr': USERNAME, 'pwd': PASSWORD}
    fp = webdriver.FirefoxProfile()
    fp.set_preference('network.proxy.type', 1)
    fp.set_preference('network.proxy.http', proxy['host'])
    fp.set_preference('network.proxy.http_port', int(proxy['port']))
    fp.set_preference('network.proxy.no_proxies_on', 'localhost, 127.0.0.1')
    fp.set_preference("network.proxy.socks_version", 5)
    credentials = '{usr}:{pwd}'.format(**proxy)
    fp.set_preference("network.proxy.type", 1)
    fp.set_preference("network.proxy.socks_version", 5)
    fp.set_preference("network.proxy.socks", HOST)
    fp.set_preference("network.proxy.socks_port", PORT)
    fp.set_preference("network.http.use-cache", False)
    driver = webdriver.Firefox(fp)
    # normally a proxy login popup should appear and we should call the login function here!!
    driver.get('https://wtfismyip.com')

Doesnt work, Any Ideas ?

Answer 1

To add foxyproxy as an extension:

driver = webdriver.Firefox(executable_path="path to geckodriver", options=options)
driver.install_addon("path to .xpi file of the extension", temporary=None)
driver.switch_to.window(driver.window_handles[1])`

So it should open a webpage where you can set your proxy. Now, to set the proxy, first i define a function that lets me insert all the proxy's stuff (host, port, username, password)

def set_proxy(address, port, usernamex, passwordx):
    proxy_address_xpath = '//*[@id="proxyAddress"]'
    proxy_address_box = driver.find_element_by_xpath(proxy_address_xpath)
    proxy_address_box.send_keys(address)
    proxy_port_xpath = '//*[@id="proxyPort"]'
    proxy_port_box = driver.find_element_by_xpath(proxy_port_xpath)
    proxy_port_box.send_keys(port)
    proxy_username_xpath = '//*[@id="proxyUsername"]'
    proxy_username_box = driver.find_element_by_xpath(proxy_username_xpath)
    proxy_username_box.send_keys(usernamex)
    proxy_pass_xpath = '//*[@id="proxyPassword"]'
    proxy_pass_box = driver.find_element_by_xpath(proxy_pass_xpath)
    proxy_pass_box.send_keys(passwordx)
    save_xpath = '/html/body/div[2]/div[2]/button[4]'
    save_box = driver.find_element_by_xpath(save_xpath)
    save_box.click()

then, with the following code, I set the proxy and select it, so that when i navigate it will use the proxy

# go to the page where you can add the proxies
xpath_back ="/html/body/div[2]/div[2]/button"
back_box = driver.find_element_by_xpath(xpath_back)
back_box.click()
    
# add proxy
add_xpath = "/html/body/div[4]/div/nav/a[1]"
add_box = WebDriverWait(driver, timeout=10).until(lambda x: x.find_element_by_xpath(add_xpath))
add_box.click()
    
# select proxy type (in my case socks5)
type_xpath = '//*[@id="proxyType"]'
type_box = driver.find_element_by_xpath(type_xpath)
type_box.click()
    
socks5_xpath = '/html/body/div[2]/div[1]/div[2]/div[1]/select/option[3]'
socks5_box = driver.find_element_by_xpath(socks5_xpath)
socks5_box.click()
    
# write the proxy stuff
set_proxy('socks5', 'host', 'port', 'pass')
    
# select the proxy you just saved (in this case it selects the first one)
change_xpath = '//*[@id="mode"]'
change_box = WebDriverWait(driver, timeout=10).until(lambda x: x.find_element_by_xpath(change_xpath))
change_box.click()
    
select_xpath = '/html/body/div[4]/div/div/div[1]/select/option[2]'
select_box = driver.find_element_by_xpath(select_xpath)
select_box.click()

then, to navigate, be sure to do that in a new windows or switch to the first one using

driver.switch_to.window(driver.window_handles[0])

so that you can keep the foxyproxy window for example to change proxy.

If you want to use other ways, check How to set proxy authentication (user & password) using Python + Selenium