Scraping web-page with button-multitems click

Question

I'am trying to scrape this web page: https://whalewisdom.com/filer/fisher-asset-management-llc#tabholdings_tab_link

I would like to setup the python selenium code, in order to setup correctly multitems in: "50" pages per page

But my code click on wrong button. where is my code error?

options = webdriver.FirefoxOptions()
options.binary_location = r'C:/Users/Mozilla Firefox/firefox.exe'
driver = selenium.webdriver.Firefox(executable_path='C:/geckodriver.exe' , options=options)
driver.execute("get", {'url': 'https://whalewisdom.com/filer/fisher-asset-management-llc#tabholdings_tab_link'})
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//label[@id='qtr-1-label']"))))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@class='btn btn-default dropdown-toggle']"))).click()

thank you for your help.

-ag

Answer 1

You code clicked on wrong button because you have multiple elements with exact same class and you are fetching the first one and clicking on it.

Also I see on the page, you sometime get a popup which may make other elements not interactable. SO we would want the popup to close first(if appeared) then move ahead.

Using Chrome driver

Setup and Imports

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

# REPLACE YOUR CHROME PATH HERE
chrome_path = r"C:\Users\hpoddar\Desktop\Tools\chromedriver_win32\chromedriver.exe"

s = Service(chrome_path)
driver = webdriver.Chrome(service=s)

Fetch the page

driver.get(' https://whalewisdom.com/filer/fisher-asset-management-llc#tabholdings_tab_link')

Close the popup(if appeared)

try:
    popup = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//a[@id='dfwid-close-184302']")))
    popup.click()
except TimeoutException:
    print("No Popup appeared on the page")

Click on dropdown and the menu item 50

dropdown = driver.find_element(By.CSS_SELECTOR, '.btn-group.dropdown')
dropdown.click()
ele50 = driver.find_element(By.XPATH, '//li[@role="menuitem"]/a[contains(text(), "50")]')
ele50.click()

Output

The above code clicks on item 50

Using Firefox driver

The imports would be same as above, the following code would also remains some with just a minute change.

# REPLACE YOUR FIREFOX DRIVER PATH HERE
firefoxpath = r'C:\Users\hpoddar\Desktop\Tools\firefoxdriver\geckodriver.exe'
s = Service(firefoxpath)
driver  = webdriver.Firefox(service=s)
driver.get(' https://whalewisdom.com/filer/fisher-asset-management-llc#tabholdings_tab_link')

try:
    popup = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//a[@id='dfwid-close-184302']")))
    popup.click()
except TimeoutException:
    print("No Popup appeared on the page")
  
dropdown = driver.find_element(By.CSS_SELECTOR, '.btn-group.dropdown')
dropdown.click()
ele50 = driver.find_element(By.XPATH, '//li[@role="menuitem"]/a[contains(text(), "50")]')
ele50.click()

Output

which similarly clicks on the desired element