CODEWITHSUNDEEP
pythonseleniumfor-loopxpathselenium-chromedriver
Wasian
Wasian

Reputation: 79

Selenium Web Scraping: Find element by text not working in script

I am working on a script to gather information off Newegg to look at changes over time in graphics card prices. Currently, my script will open a Newegg search on RTX 3080's through Chromedriver and then click on the link for Desktop Graphics Cards to narrow down my search. The part that I am struggling with is developing a for item in range loop that will let me iterate through all 8 search result pages. I know that I could do this by simply changing the page number in the URL, but as this is an exercise that I'm trying to use to learn Relative Xpath better, I want to do it using the Pagination buttons at the bottom of the page. I know that each button should contain inner text of "1,2,3,4 etc." but whenever I use text() = {item} in my for loop, it doesn't click the button. The script runs and doesn't return any exceptions, but doesn't do what I want it too. Below I have attached the HTML for the page as well as my current script. Any suggestions or hints are appreciated.

enter image description here

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import pandas as pd
import time

options = Options()

PATH = 'C://Program Files (x86)//chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.newegg.com/p/pl?d=RTX+3080'

driver.maximize_window()
driver.get(url)

card_path = '/html/body/div[8]/div[3]/section/div/div/div[1]/div/dl[1]/dd/ul[2]/li/a'
desktop_graphics_cards = driver.find_element(By.XPATH, card_path)
desktop_graphics_cards.click()
time.sleep(5)

graphics_card = []
shipping_cost = []
price = []
total_cost = []

for item in range(9):
    try:
        #next_page_click = driver.find_element(By.XPATH("//button[text() = '{item + 1}']"))
        print(next_page_click)
        next_page_click.click()
    except:
        pass

Upvotes: 1

Views: 252

Answers (1)

Prophet
Prophet

Reputation: 33361

The pagination buttons are out of the initially visible area.
In order to click these elements you will have to scroll the page until the element appears.
Also, you will need to click next page buttons starting from 2 up to 9 (including) while you trying to do this with numbers from 1 up to 9.
I think this should work better:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import pandas as pd
import time

options = Options()

PATH = 'C://Program Files (x86)//chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.newegg.com/p/pl?d=RTX+3080'
actions = ActionChains(driver)

driver.maximize_window()
driver.get(url)

card_path = '/html/body/div[8]/div[3]/section/div/div/div[1]/div/dl[1]/dd/ul[2]/li/a'
desktop_graphics_cards = driver.find_element(By.XPATH, card_path)
desktop_graphics_cards.click()
time.sleep(5)

graphics_card = []
shipping_cost = []
price = []
total_cost = []

for item in range(2,10):
    try:
        next_page_click = driver.find_element(By.XPATH(f"//button[text() = '{item}']"))
        actions.move_to_element(next_page_click).perform()
        time.sleep(2)
        #print(next_page_click) - printing a web element itself will not give you usable information
        next_page_click.click()
        #let the next page loaded, it takes some time
        time.sleep(5)
    except:
        pass

Upvotes: 1

Related Questions