CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

Can't move on to the next page

I've written a script in python with selenium to traverse different pages staring from the first page through pagination. However, there are no options for next page button except for some numbers. When i click on that number it takes me to the next page. Anyways, when i try to do that with my script, it does click on the second page and goes there but it doesn't slide anymore, I meant instead of going on to the third page it breaks throwing the following error.

line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

Script I'm trying with:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("http://www.cptu.gov.bd/AwardNotices.aspx")
wait = WebDriverWait(driver, 10)
driver.find_element_by_id("imgbtnSearch").click()
for item in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#dgAwards > tbody > tr > td > a"))):
    item.click()
driver.quit()

Elements within which the pagination numbers are:

<tr align="right" valign="top" style="font-size:XX-Small;font-weight:normal;white-space:nowrap;">
        <td colspan="8"><span>Page: </span><a href="javascript:__doPostBack('dgAwards$ctl01$ctl01','')">1</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl02','')">2</a>&nbsp;<span>3</span>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl04','')">4</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl05','')">5</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl06','')">6</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl07','')">7</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl08','')">8</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl09','')">9</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl10','')">10</a>&nbsp;<a href="javascript:__doPostBack('dgAwards$ctl01$ctl11','')">...</a></td>
    </tr>

Btw, pagination option appears upon clicking on "search" button in the main page.

Upvotes: 3

Views: 378

Answers (1)

Andersson
Andersson

Reputation: 52685

You cannot iterate through the list of pre-defined elements because after you make a click() page refreshes and those elements become stale

You can try below:

from selenium.common.exceptions import NoSuchElementException    

page_counter = 2
while True:
    try:
        if not page_counter % 10 == 1:
            driver.find_element_by_link_text(str(page_counter)).click()
            page_counter += 1
        else:
           driver.find_elements_by_link_text("...")[-1].click() 
           page_counter += 1
    except NoSuchElementException :
        break

This should allow you to switch to next page while it's possible

Upvotes: 3

Related Questions