CODEWITHSUNDEEP
pythonselenium
Prabhjot Singh Rai
Prabhjot Singh Rai

Reputation: 2515

How to scroll to the end of the page using selenium in Python?

I am trying to scroll to the end of a page so that I can make all the data visible and extract it. I tried to find a command for it but it's available in java (driver.executeScript) but couldn't find for python. Right now I am making the computer press the end key thousand times:

while i<1000:
    scroll = driver.find_element_by_tag_name('body').send_keys(Keys.END)
    i+=1

And I also tried driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") but it scrolls to the end of the loaded page and the same thing END key does. Once at the bottom of the page, next content loads. But now it doesn't scroll again.

I know there will be a very nice alternative for this.

How do I scroll to the end of the page using selenium in Python?

Upvotes: 28

Views: 60246

Answers (7)

Julian
Julian

Reputation: 152

To add some more context, in some cases you'd neither know the element you're waiting to load, nor you'd know the number of iterations (scrolls) needed to reach the bottom of the page. What I found out to be a useful case in such scenarios is just take the entire page source, and while it changes on every scroll, you'd get some difference every time, regardless on its contents. Here's my suggestion:

html = driver.find_element(By.TAG_NAME, 'html')
while True:
    s1 = driver.page_source
    time.sleep(random.randint(1, 3))
    html.send_keys(Keys.PAGE_DOWN)
    html.send_keys(Keys.END)
    time.sleep(random.randint(1, 3))
    html.send_keys(Keys.END)
    s2 = driver.page_source
    if s1==s2:
        break

Eventually, I'd delete s1 and s2 - large unneeded chunks of text.

Upvotes: 0

denis hnatiuk
denis hnatiuk

Reputation: 11

#go to a the element that actually scrolls like a tbody
element_in_table = self.driver.find_element(By.XPATH, html_tbody_path)
             
ActionChains(self.driver).move_to_element(element_in_table).perform()
element_in_table.click()
self.driver.execute_script("window.scrollTo(0, 
document.body.scrollHeight);var 
lenOfPage=document.body.scrollHeight;return lenOfPage;")

Upvotes: 0

Nitin Kumar
Nitin Kumar

Reputation: 69

Since there is no link provided for the website I am going to assume that there is some kind of See More/Load More clickable element present on the page. Here is what I like to and its pretty simple.

count=10000
while count>1:
   try:
       button=driver.find_element_by_xpath('//*[@id="load_more"]')
       button.click()
       count-=1
       time.sleep(2)
   except StaleElementReferenceException:
       button=driver.find_element_by_xpath('//*[@id="load_more"]')
       button.click()
       time.sleep(2)

Upvotes: 0

user53558
user53558

Reputation: 367

None of these were working for me, but the below solution did:

driver.get("https://www.youtube.com/user/teachingmensfashion/videos")


def scroll_to_bottom(driver):

    old_position = 0
    new_position = None

    while new_position != old_position:
        # Get old scroll position
        old_position = driver.execute_script(
                ("return (window.pageYOffset !== undefined) ?"
                 " window.pageYOffset : (document.documentElement ||"
                 " document.body.parentNode || document.body);"))
        # Sleep and Scroll
        time.sleep(1)
        driver.execute_script((
                "var scrollingElement = (document.scrollingElement ||"
                " document.body);scrollingElement.scrollTop ="
                " scrollingElement.scrollHeight;"))
        # Get new position
        new_position = driver.execute_script(
                ("return (window.pageYOffset !== undefined) ?"
                 " window.pageYOffset : (document.documentElement ||"
                 " document.body.parentNode || document.body);"))

scroll_to_bottom(driver)

Upvotes: 17

0x48piraj
0x48piraj

Reputation: 417

You can utilize scrollingElement with scrollTop and scrollHeight to to scroll to the end of a page.

driver.execute_script("var scrollingElement = (document.scrollingElement || document.body);scrollingElement.scrollTop = scrollingElement.scrollHeight;")

References :

  1. Scroll Automatically to the Bottom of the Page
  2. Document.scrollingElement - Web APIs | MDN
  3. Element.scrollHeight - Web APIs | MDN
  4. Element.scrollTop - Web APIs | MDN

Upvotes: 6

Alex
Alex

Reputation: 12913

This can be done in one line by scrolling to document.body.scrollHeight

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Upvotes: 23

Prabhjot Singh Rai
Prabhjot Singh Rai

Reputation: 2515

Well I finally figured out a solution:

lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
    match=False
        while(match==False):
                lastCount = lenOfPage
                time.sleep(3)
                lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
                if lastCount==lenOfPage:
                    match=True

Upvotes: 32

Related Questions