CODEWITHSUNDEEP
pythonseleniumweb-scraping
sir_chocolate_soup
sir_chocolate_soup

Reputation: 449

Getting next page data through selenium and request

I'm looking to click on the next page to get the listed product data: https://www.riteaid.com/shop/grocery

When I look at the HTML element, the Next button is actually not a button element, but a span tag. Looking at the developer tools, I noticed that there is a tag that dynamically changes when I click on the next button without moving to a different url.

<div data-pager="p:pg1" data-tp="89" data-cp="2">

How would one go about getting the next page data or send the request to get the next page data? I thought I'd need selenium since it is interactive, but I got the error "selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable".

Thanks!

Upvotes: 0

Views: 1085

Answers (2)

Arundeep Chohan
Arundeep Chohan

Reputation: 9969

You can invoke click onto the element.

driver.get("https://www.riteaid.com/shop/grocery")
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.presence_of_element_located((By.XPATH, "//span[text()='Next']/parent::div")))
driver.execute_script("arguments[0].click();", elem)

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

Upvotes: 1

Saad Saadi
Saad Saadi

Reputation: 1071

I checked the site and you can navigate to next page like this:

next_page_xpath = "//div[@class='rfk_next']"
driver.find_element_by_xpath(next_page_xpath).click()

Do remember to do while and put above code in try/except, so that it breaks from while loop once there will be no next page.

Upvotes: 0

Related Questions