CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverinfinite-scroll
Eddyhut
Eddyhut

Reputation: 55

Python Selenium not scrolling down infinite page

I'm having some trouble scrolling down an infinite scrolling page using Python Selenium. I'd like to scroll down this page so that I can click on to each film on the page. The code runs to the end and quits the driver but without scrolling down the page - what's wrong?

def scroll(driver):
    timeout = 5

    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # load the website
        time.sleep(5)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")

driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
scroll(driver)
driver.quit()

Upvotes: 1

Views: 389

Answers (2)

PDHide
PDHide

Reputation: 19929

driver = webdriver.Chrome()


def scroll(driver):
    timeout = 5
    time.sleep(5)
    # Get scroll height
    scrollelement = driver.execute_script(
        "return document.querySelector('.tabs-inner [tab=\"popular\"] ion-content').shadowRoot.querySelector('[class=\"inner-scroll scroll-y\"]')")
    last_height = driver.execute_script(
        "return arguments[0].scrollHeight", scrollelement)
    new_height=0
    print(new_height, last_height)
    while True:
        # Scroll down to bottom
       
        driver.execute_script(
            "arguments[0].scrollTo(0, arguments[0].scrollHeight);", scrollelement)

        # load the website
        time.sleep(5)

        # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script(
            "return arguments[0].scrollHeight", scrollelement)
        print(new_height,last_height)


driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
scroll(driver)
driver.quit()

You should do the scroll on the scrollelement as the scrollbar is also an element in the page. But the scrollbar is in side shadowRoot so use execute_script

Upvotes: 1

Vova
Vova

Reputation: 3543

Try to use this:

import time
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.justwatch.com/uk/provider/bfi-player/movies")
time.sleep(3)
driver.execute_script("document.body.getElementsByTagName('ion-content')[0].scrollToBottom();")
time.sleep(3)
driver.quit()

Upvotes: 1

Related Questions