CODEWITHSUNDEEP
python-3.xselenium-webdriverxpathwebdriverwaitdopostback
Hosam Gamal
Hosam Gamal

Reputation: 183

How to paginate through the page numbers when href contains javascript:__doPostBack()

I'm trying to scrape this website http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae but when I want to go to next page I can't because the link doesn't change you will find that pages links are like that 

href="javascript:__doPostBack('sb$grd','Page$1')"

I have a code that I tried but it only goes to page 2 and then gave me an error: tale element reference: element is not attached to the page document

from selenium import webdriver
url = 'http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae'
driver = webdriver.Chrome()
driver.get(url)
btn = [w for w in driver.find_elements_by_xpath('//*[@id="sb_grd"]/tbody/tr[26]/td/table/tbody/tr/td/a')]
for b in btn:
    driver.execute_script("arguments[0].click();", b)

Upvotes: 1

Views: 931

Answers (2)

undetected Selenium
undetected Selenium

Reputation: 193338

To paginate through the page numbers with href attribute as "javascript:__doPostBack('sb$grd','Page$2')" you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("http://www.mfa.gov.tr/sub.ar.mfa?dcabec54-44b3-4aaa-a725-70d0caa8a0ae")
while True:
    try:
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//table[@id='sb_grd']//table/tbody/tr//td/span//following::td[1]/a"))).click()
        print("Next page clicked")
    except TimeoutException:
        print("No more pages")
        break
driver.quit()

  • Console Output:

    Next page clicked
Next page clicked
Next page clicked
.
.
.
No more pages

Reference

You can find a relevant detailed discussion in:

Upvotes: 1

SimonR
SimonR

Reputation: 523

You can see that a request is made under the wood to this url : http://www.mfa.gov.tr/default.fr.mfa Do F12 and go to the network tab to see that.

All the articles you need will be in the document at this url, you don't need to paginate as the pagination has just a purpose display. No request is being made by this pagination, expect to download the displayed image.

Upvotes: 0

Related Questions