CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

Trouble dealing with next page link

I've written a script in python in combination with selenium to get some data from a javascript enabled webpage. There are three things to do before clicking on the next page button because the next page link appears only when the webpage contains the search results. The three things are: filling in two search boxes and clicking on search button. However, my script can do the three things flawlessly but breaks when it is supposed to click on the next page link (throwing timeout exception). As you can see, I've already tried with three different options to click on the next page link but never succeeded. The rest two I tried with are commented out. What to do to click on the next page button successfully?

The script I've tried with:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get("https://brokercheck.finra.org/")

wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[placeholder='Name or CRD#']"))).send_keys("Michael John")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[placeholder='Firm Name or CRD# (optional)']"))).send_keys("Morgan Stanley")
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'.md-button'))).click()
# wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.pagination-next a'))).click()
# wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'.pagination-next a'))).click()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'.pagination-next a'))).click()
driver.quit()

Elements within which next page links are:

<ul class="pagination ng-pristine ng-untouched ng-valid ng-scope ng-isolate-scope" data-ng-if="listCtrl.getTotalResults()" total-items="listCtrl.getDisplayResults()" ng-model="listCtrl.currentPage" max-size="1" page-label="listCtrl.pageLabel($page)" items-per-page="listCtrl.itemsPerPage" ng-change="listCtrl.pageChanged()" boundary-links="true" previous-text="‹" next-text="›" first-text="«" last-text="»" aria-invalid="false">
  <!-- ngIf: ::boundaryLinks --><li ng-if="::boundaryLinks" ng-class="{disabled: noPrevious()||ngDisabled}" class="pagination-first ng-scope disabled"><a href="" ng-click="selectPage(1, $event)" class="ng-binding">«</a></li><!-- end ngIf: ::boundaryLinks -->
  <!-- ngIf: ::directionLinks --><li ng-if="::directionLinks" ng-class="{disabled: noPrevious()||ngDisabled}" class="pagination-prev ng-scope disabled"><a href="" ng-click="selectPage(page - 1, $event)" class="ng-binding">‹</a></li><!-- end ngIf: ::directionLinks -->
  <!-- ngRepeat: page in pages track by $index --><li ng-repeat="page in pages track by $index" ng-class="{active: page.active,disabled: ngDisabled&amp;&amp;!page.active}" class="pagination-page ng-scope active"><a href="" ng-click="selectPage(page.number, $event)" class="ng-binding">1 of 27 pages</a></li><!-- end ngRepeat: page in pages track by $index -->
  <!-- ngIf: ::directionLinks --><li ng-if="::directionLinks" ng-class="{disabled: noNext()||ngDisabled}" class="pagination-next ng-scope"><a href="" ng-click="selectPage(page + 1, $event)" class="ng-binding">›</a></li><!-- end ngIf: ::directionLinks -->
  <!-- ngIf: ::boundaryLinks --><li ng-if="::boundaryLinks" ng-class="{disabled: noNext()||ngDisabled}" class="pagination-last ng-scope"><a href="" ng-click="selectPage(totalPages, $event)" class="ng-binding">»</a></li><!-- end ngIf: ::boundaryLinks -->
</ul>

Upvotes: 1

Views: 369

Answers (1)

Andersson
Andersson

Reputation: 52665

There are 2 paginations on page with the same locators: at the top and at the bottom.

To handle top you need to execute driver.maximize_window() to make it visible and then use same code as you tried:

link = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'.pagination-next a')))
driver.execute_script('arguments[0].scrollIntoView();', link)
link.click()

To handle bottom pagination:

wait.until(EC.visibility_of_element_located((By.XPATH,'(//*[contains(@class, "pagination-next")]//a)[2]'))).click()

Upvotes: 1

Related Questions