CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingbeautifulsoup
sputnikk1093
sputnikk1093

Reputation: 43

How to identify the classname or id in Python Scraping with beautifulsoup and selenium

I am building a scraper code and already have been able to read the table and the information that I want. The problem is with the next page link, I have tried using a class name and also and svg tag but the code breaks as the value of the class name changes.

Here is the link of the page

Page to scrape

And the code for the element css that the code runs to click for next page is this

driver.find_element_by_css_selector('#root > div > div > main > div.ez6st4XksKUGZfGdvIhjV > section > div:nth-child(1) > div._1c5cPqlj4FoguvpKXSY69p > div > span:nth-child(3) > svg').click()

It seems that when the value of the class name changes it breaks and changes the element to click and I havent found a way to repeat without changing the element in order to repeat for multiple pages with the same structure.

Thanks

Upvotes: 4

Views: 858

Answers (2)

Andersson
Andersson

Reputation: 52685

You can use below line to click Next button without referring to dynamic class names:

driver.find_element_by_xpath('//span[@value]/following-sibling::span/*[name()="svg"]').click()

The same with CSS-selector:

driver.find_element_by_css_selector('span[value] + span > svg')

Update

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

while True:
    try:
        wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'span[value] + span > svg'))).click()
    except:
        break

Upvotes: 0

QHarr
QHarr

Reputation: 84465

As you can click the span you can also use

from selenium import webdriver
d  = webdriver.Chrome()
url = 'https://super.walmart.com.mx/despensa/enlatados-y-conservas/chiles-enlatados/_/N-10kldy7?%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3F%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3F%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3FNs=product.displayText%7C0&offSet=0&storeId=0000009999&No=40'
d.get(url)
# example number of clicks below
for i in range(2):
    d.find_element_by_xpath("//*[starts-with(@d,'M0')]/parent::*/parent::span").click()

Upvotes: 2

Related Questions