CODEWITHSUNDEEP
proxyselenium-chromedriverbasic-authenticationheadlesschrome-options
Solomon Raja
Solomon Raja

Reputation: 1810

Chrome headless browser with corporate proxy authetication not working

I am trying to run a Chrome headless browser sitting behind a corporate proxy. I tried below code. But unable to pass through it.

public class HeadlessChrome 
{
    WebDriver driver;

    @Test
    public void createChromeDriverHeadless() throws InterruptedException
    {
        System.setProperty("webdriver.chrome.driver", "D:\\LocalData\\workspace\\Drivers and Libraries\\driver\\chromedriver.exe");

        ChromeOptions chromeOptions = new ChromeOptions();

        Proxy proxy = new Proxy();
        proxy.setHttpProxy("http://user:pwd@server:port");
        proxy.setSslProxy("http://user:pwd@server:port");

//      chromeOptions.setCapability("proxy", proxy);
        chromeOptions.addArguments("--proxy-server=user:pwd@server:port");
        chromeOptions.addArguments("--headless");
        chromeOptions.addArguments("--disable-gpu");
        chromeOptions.addArguments("start-maximized");  

        driver = new ChromeDriver(chromeOptions);
        driver.get("http://seleniumhq.org");       

        Thread.sleep(5000);
        System.out.println("Title : " + driver.getTitle());
        assertTrue(driver.findElement(By.id("q")).isDisplayed());
        driver.quit();
    }
}

Please help me out.

Upvotes: 3

Views: 3436

Answers (3)

Pavel
Pavel

Reputation: 165

It's easy to achieve with selenium 4 (currently in beta). You can do it in multiple ways:

  1. You basically need to register a check for whether to apply the credentials for any request for authorization. Works for both - basic and proxy auth popups.

    ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";
//register our check here
driver.register(UsernameAndPassword.of(USER_NAME, PASSWORD));
driver.get("https://jigsaw.w3.org/HTTP/");
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

  2. Using CDP Network domain. Doesn't work for proxy authorization popup (for example here is the similar issue in puppeteer which goes down to the chrome project)

    ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";

DevTools devTools = driver.getDevTools();
//create a cdp session
devTools.createSession();
//enable network first
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
//Open website
driver.get("https://jigsaw.w3.org/HTTP/");
//Create and send the authorization header
Map<String, Object> headers = new HashMap<>();
String basicAuth = "Basic " + new String(Base64.getEncoder().encode(String.format("%s:%s", USER_NAME, PASSWORD).getBytes()));
headers.put("Authorization", basicAuth);
devTools.send(Network.setExtraHTTPHeaders(new Headers(headers)));
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

  3. Using the CDP Fetch domain. Works for both - basic and proxy auth popups.

    ChromeDriver driver = new ChromeDriver(new ChromeOptions().setHeadless(true));
String USER_NAME = "guest";
String PASSWORD = "guest";

DevTools devTools = driver.getDevTools();
//create a cdp session
devTools.createSession();
//enable Fetch first
devTools.send(Fetch.enable(Optional.empty(), Optional.of(true)));
devTools.addListener(Fetch.requestPaused(), requestPaused -> devTools.send(Fetch.continueRequest(requestPaused.getRequestId(), Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())));
devTools.addListener(Fetch.authRequired(), authRequired -> devTools.send(Fetch.continueWithAuth(authRequired.getRequestId(), new AuthChallengeResponse(PROVIDECREDENTIALS, Optional.of(USER_NAME), Optional.of(PASSWORD)))));

//Open website
driver.get("https://jigsaw.w3.org/HTTP/");
//Click on the link to show an authentication popup
driver.findElement(By.linkText("Basic Authentication test")).click();
String msg = driver.findElement(By.tagName("html")).getText();
assert msg.equalsIgnoreCase("Your browser made it!");

Upvotes: 1

Tal Zilberman
Tal Zilberman

Reputation: 11

There is headless browser called Linken-sphere who cooperates with Luminati. They some good offers . you should check them up. https://miped.ru/f/threads/linken-sphere-antidetekt-brauzer-novogo-pokolenija.67098/

Upvotes: 1

Tarun Lalwani
Tarun Lalwani

Reputation: 146630

If you were not using headless you could have used the approach in below link

user:pass proxies with selenium

But with headless extension are currently not allowed. So now your option is add another proxy

chrome -> (intermediate proxy w/o auth) -> corporate proxy w/ auth -> internet

One options is to use polipo

https://www.irif.fr/~jch/software/polipo/

with below config

parentAuthCredentials=username:password
parentProxy=corporateproxy:port

and then use

chromeOptions.addArguments("--proxy-server=http://polipoproxy:port");

The default would be 127.0.0.1:8123 in don't override in polipo config.

Other options you can use

  • Use squid proxy instead of polipo

  • Write your own proxy forwarder using python or node or any other language you are comfortable with

Upvotes: 4

Related Questions