CODEWITHSUNDEEP
pythonhtmlseleniumxpathweb-crawler
paaaaat
paaaaat

Reputation: 91

Selenium click on a next-page link not loading the next page

I'm new to selenium and webscraping and I'm trying to get information from the link: https://www.carmudi.com.ph/cars/civic/distance:50km/?sort=suggested

Here's a snippet of the code I'm using:

while max_pages > 0:
                results.extend(extract_content(driver.page_source))
                next_page = driver.find_element_by_xpath('//div[@class="next-page"]')
                driver.execute_script('arguments[0].click();', next_page)
                max_pages -= 1

When I try to print results, I always get (max_pages) of the same results from page 1. The "Next page" button is visible in the page and when I try to find elements of the same class, it only shows 1 element. When I try getting the element by the exact xpath and performing the click action on it, it doesn't work as well. I enclosed it in a try-except block but there were no errors. Why might this be?

Upvotes: 0

Views: 1890

Answers (2)

JeffC
JeffC

Reputation: 25610

You are making this more complicated than it needs to be. There's no point in using JS clicks here... just use the normal Selenium clicks.

while True:
    # do stuff on the page
    next = driver.find_element_by_css_selector("a[title='Next page']")
    if next
        next.click()
    else
        break

Upvotes: 1

Eduard Florinescu
Eduard Florinescu

Reputation: 17531

replace:

next_page = driver.find_element_by_xpath('//div[@class="next-page"]')
driver.execute_script('arguments[0].click();', next_page)

with:

driver.execute_script('next = document.querySelector(".next-page"); next.click();')

If you try next = document.querySelector(".next-page"); next.click(); in console you can see it works.

Upvotes: 0

Related Questions