CODEWITHSUNDEEP
pythonseleniumgoogle-chromeselenium-webdriverautomation
Ajay Kumar
Ajay Kumar

Reputation: 1343

Selenium can't open a second page

I am using Selenium to open different pages of a site. Have tried multiple times but the browser does not open a second webpage after the initial GET call. Have tried on both Chrome and Safari. Here is my code:

driver = webdriver.Chrome()
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.set_page_load_timeout(30)
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

Here is the error I get for the second call:

enter image description here

The info from Network logs is Error 504, but I have verified that it works perfectly when done on another window of the browser, without automation

Upvotes: 1

Views: 5426

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193088

A bit of more information about your usecase would have helped to construct a more canonical answer. However I was able to access the Page 2 of justdial.com/Chennai/Hr-Consultancy-Services with a minimized code block as follows:

  • Code Block:

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")

  • Browser Snapshot:

justdial

But while sending multiple get() one after another:

driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

It seems ChromeDriver initiated Chrome Browser gets detected and the following error is shown:

An error occurred while processing your request.
Reference #97.e5732c31.1612205693.6fd2708

Solution

To avoid the detection you can add the following option:

--disable-blink-features=AutomationControlled

Example

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

Upvotes: 2

PDHide
PDHide

Reputation: 19929

options.add_experimental_option(
    "excludeSwitches", ['enable-automation'])

options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
options.add_argument("--remote-debugging-port=9222")

driver = webdriver.Chrome(options=options)
driver.get(
    "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.set_page_load_timeout(30)
driver.get(
    "https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

the website is detecting automation use above code :)

You can also do this in single line

Just add below argument:

options.add_argument('--disable-blink-features=AutomationControlled')

disabling enable-automation , or disabling automation controller disables webdriver.navigator which few websites uses to detect automation scripts

Upvotes: 3

Related Questions