CODEWITHSUNDEEP
javascriptjavaselenium
Johannes Mols
Johannes Mols

Reputation: 1041

Selenium: Scroll to end of page in dynamically loading webpage

I have a webpage that keeps loading new items when scrolling down the page until every item is loaded.

I'm working with Selenium in Java, and need to scroll down to the bottom of the page in order to load everything.

I have tried several different options, like scrolling to an element of the bottom of the page:

WebElement copyrightAtEndOfPage = webDriver.findElement(By.xpath("//a[@href='/utils/copyright.html']"));
((JavascriptExecutor) webDriver).executeScript("arguments[0].scrollIntoView();", copyrightAtEndOfPage);

This only scrolls down once though, and then the webpage keeps loading.

I also tried this approach, which also only scrolls down once, because it only takes the browser height into consideration.

Any help is highly appreciated.

Upvotes: 29

Views: 41844

Answers (11)

user2427306
user2427306

Reputation: 39

modal_elem = browser.find_element(By.CLASS_NAME,'yourClassName')`
def scroll_down(self):
"""A method for scrolling the page."""

    # Get scroll height.
    last_height = self.execute_script("return arguments[0].scrollHeight", modal_elem)
    print('last_height')
    print(last_height)
 while True:

    # Scroll down to the bottom.
    self.execute_script("arguments[0].scrollTo(0, arguments[0].scrollHeight);", modal_elem)
    
    # Wait to load the page.
    time.sleep(2)
    
    # Calculate new scroll height and compare with last scroll height.
    new_height = self.execute_script("return arguments[0].scrollHeight", modal_elem)
    print('new_height')
    print(new_height)
    if new_height == last_height:

        break

    last_height = new_height
    
 scroll_down(browser)

Upvotes: 0

Mahmud Oly
Mahmud Oly

Reputation: 1

to scroll and wait to load more property till end

                lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                match = False
                while not match:
                    lastCount = lenOfPage
                    time.sleep(2)
                    lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                    if lastCount == lenOfPage:
                        match = True

Upvotes: 0

indayush
indayush

Reputation: 67

You can add the following code to keep pressing page Down button:

new Actions(driver).sendKeys(Keys.PAGE_DOWN).perform();

Upvotes: 0

ukod
ukod

Reputation: 149

C# version of the Ratmir Asanov's answer:

var lastHeight =  driver.ExecuteScript("returndocument.body.scrollHeight");
while (true)
{
    driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
    await Task.Delay(500);

    var newHeight = driver.ExecuteScript("return document.body.scrollHeight");
    Console.WriteLine(lastHeight + " - " + newHeight);
    if (newHeight.Equals(lastHeight))
        break;

    lastHeight = newHeight;
}

Upvotes: 2

Nayanjyoti Rabha
Nayanjyoti Rabha

Reputation: 11

Making a slight correction to the above stated answers. The variable 'start' of type long keeps changing after every scroll and the value becomes same after it reaches the end of the webpage. And as it an infinite loop with will keep returning the same value again and again. So, I just took the 'temp' variable and checked two consecutive values are same or not as the values remain same after the end is reached. As soon at it finds the same it exits the loop.

    try {
            long temp = 0;
            while (true) {
                ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
                Thread.sleep(2500);
                long start = (Long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                if (start == temp) {
                    break;
                }
                temp = start;
            }
            System.out.println("completed the scroll");
        } catch (Exception e) {
            e.printStackTrace();
        }

Upvotes: 1

MdaCosta
MdaCosta

Reputation: 49

Updated code that worked for me:

try {
                    long lastHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                    int cont=1000;
                    while (true) {
                        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, "+cont+");");
                        Thread.sleep(2000);

                        long newHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                        if (newHeight <= cont) {
                            break;
                        }
//                      lastHeight = newHeight;
                        cont+=500;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

Upvotes: 0

buggy_code312
buggy_code312

Reputation: 11

I found another solution to the dynamically loading page.

Count the elements that are displayed every scroll before and after the scroll and compare them to determine if you've scrolled to the bottom.

var reachedEnd = false;
oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

while (!reachedEnd)
{
    driver.FindElement(By.CssSelector("body")).SendKeys(Keys.End);
    Thread.Sleep(500);
    oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

    if (newCount == oldCount)
    {
        reachedEnd = true;
    }
    else
    {
        newCount = oldCount;
    }
}

Upvotes: 0

Zenab Gorach
Zenab Gorach

Reputation: 41

Updating the above solution by Prabhat further as it was still giving me compilation error.

    try {
        Object lastHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");

        while (true) {
            ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
            Thread.sleep(2000);

            Object newHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
            if (newHeight.equals(lastHeight)) {
                break;
            }
            lastHeight = newHeight;
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

Upvotes: 0

Prabhat Arya
Prabhat Arya

Reputation: 39

Updated Johannes code a bit to make it functional.

JavascriptExecutor js = (JavascriptExecutor) driver;
try {
    long lastHeight=((Number)js.executeScript("return document.body.scrollHeight")).longValue();
    while (true) {
        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = ((Number)js.executeScript("return document.body.scrollHeight")).longValue();
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

Upvotes: 3

Ratmir Asanov
Ratmir Asanov

Reputation: 6459

I will provide you code in Python for this. I think it's easy to translate to Java:

def scroll_down(self):
    """A method for scrolling the page."""

    # Get scroll height.
    last_height = self.driver.execute_script("return document.body.scrollHeight")

    while True:

        # Scroll down to the bottom.
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load the page.
        time.sleep(2)

        # Calculate new scroll height and compare with last scroll height.
        new_height = self.driver.execute_script("return document.body.scrollHeight")

        if new_height == last_height:

            break

        last_height = new_height

Hope it helps you!

Upvotes: 63

Johannes Mols
Johannes Mols

Reputation: 1041

Thanks to Ratmir Asanov (see the approved answer above), I translated the Python code into Java to make it easier to implement for other people.

try {
    long lastHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");

    while (true) {
        ((JavascriptExecutor) webDriver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

Upvotes: 13

Related Questions