How To Scroll Inside An Element On A Webpage (Selenium Python)

Question

How can I scroll down in a certain element of a webpage in Selenium?

Basically my goal is to scroll down in this element until new profile results stop loading.

Let's say that there should be 100 profile results that I'm trying to gather.

By default, the webpage will load 30 results.

I need to scroll down IN THIS SECTION, wait a few seconds for 30 more results to load, repeat (until all results have loaded).

I am able to count the number of results with:

len(driver.find_elements(By.XPATH, "//div[@class='virtual-box']"))

I already have all the other code written, I just need to figure out the line of code to get Selenium to scroll down like 2 inches.

I've looked around a bunch and can't seem to find a good answer (that or I suck at googling).

This is a section of my code:

(getting the total number of profiles currently on the page = max_prof)

while new_max_prof > max_prof:
    scroll_and_wait(profile_number)

    if max_prof != new_max_prof:  # to make sure that they are the same
        max_prof = new_max_prof

...and here is the function that it is calling (which currently doesn't work because I can't get it to scroll)

def scroll_and_wait(profile_number=profile_number):  # This doesn't work yet
    global profile_xpath
    global new_max_prof
    global max_prof
    print('scrolling!')

    #driver.execute_script("window.scrollTo(0,1080);") #  does not work

    temp_xpath = profile_xpath + str(max_prof) + ']'
    element = driver.find_element(By.XPATH, temp_xpath)
    ActionChains(driver).scroll_to_element(element).perform()  # scrolls to the last profile
    element.click()  # selects the last profile

    # Tested and this does not seem to load the new profiles unless you scroll down.

    print('did the scroll!!!')
    time.sleep(5)

    new_max_prof = int(len(driver.find_elements(By.XPATH, "//div[@class='virtual-box']")))
    print('new max prof is: ' + str(new_max_prof))
    time.sleep(4)

I tried:

#1. driver.execute_script("window.scrollTo(0,1080);") and driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")``` but neither seemed to do anything.

#2. ActionChains(driver).scroll_to_element(element).perform() hoping that if I scrolled to the last profile on the page, it would load the next one (it doesn't)

#3. Using pywin32 win32api.mouse_event(MOUSEEVENTF_WHEEL, -300, 0) to simulate the mouse scrolling. Didn't seem to work, but even if it did, I'm not sure this would solve it because it would really need to be in the element of the webpage. Not just going to the bottom of the webpage.

Answer 1

So basically your requirement is to scroll a particular section of a webpage.

While most of the available answers revolve around scrolling the complete web page itself, it is indeed possible to scroll a particular segment of the webpage entirely.

Although @lukeprofits did provide a perfectly working solution, there is a limitation to it since there are fair number of sites where the scrollbar is not an identifiable element in source code and therefore clicked to scroll the section down.

What you can do in such a case is to manipulate the section using javascript.

Firstly identify the sub-section of the webpage that you want to scroll

xpath_element = "//aside[@class='sidebar mCustomScrollbar_mCS_2']//div[@class='mCSB_container']"

section = driver.find_element(By.XPATH, xpath_element)

Now for an example, let's say you want to scroll the particular section 3 times, therefore we can initialize a counter variable for the same:

 counter = 0



while counter < 3:  # this will scroll 3 times
     driver1.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + arguments[0].offsetHeight;',
                                 section)
      counter += 1

      # add a timer for the data to fully load once you have scrolled the section

      time.sleep(5) # You might need to install time library to use this statement

And this will help you scroll the section in your webpage. Please note that you can modify the execute_script() as per your requirement, so for example if you wish to scroll to the bottom of the section in each attempt, you can simply use:

driver.execute_script(
    'arguments[0].scrollTop = arguments[0].scrollTop + document.getElementById("search-results-container").scrollHeight;',
    section,
)

where you are getting the scrolling height of section that you were trying to scroll.

Answer 2

OKAY! I found something that works. (If anyone knows a better solution please let me know)

You can use this code to scroll to the bottom of the page:

driver.find_element(By.TAG_NAME, 'html').send_keys(Keys.END)  # works, but not inside element.

What I had to do was more complicated though (since I am trying to scroll down IN AN ELEMENT on the page, and not just to the bottom of the page).

IF YOUR SCROLL BAR HAS ARROW BUTTONS at the top/buttons, try just clicking them with .click() or .click_and_hold() that's a much easier solution that trying to scroll and does the same thing.

IF, LIKE ME, YOUR SCROLL BAR HAS NO ARROW BUTTONS, you can still click on the scroll bar path at the bottom/top and it will move. If you find the XPATH to your scrollbar, then click it, it will click in the middle (not helpful), but you can offset this on the x/y axis with ".move_by_offset(0, 0)" so for example:

# import ActionChains
from selenium.webdriver.common.action_chains import ActionChains

scroll_bar_xpath = "//div[@ng-if='::vm.isVirtual']/div[@class='ps-scrollbar-y-rail']"
element = driver.find_element(By.XPATH, scroll_bar_xpath)

# Do stuff
ActionChains(driver).move_to_element(element).move_by_offset(0,50).click().perform()

Now normally, you wouldn't want to use a fixed pixel amount (50 on the y axis) because if you change the browser size, or run the program on a different monitor, it could mess up.

To solve this, you just need to figure out the size of the scroll bar, so that you know where the bottom of it is. All you have to do is:

element = driver.find_element(By.XPATH, scroll_bar_xpath)
size = element.size
w = size['width']
h = size['height']\
print('size is: ' + size)
print(h)
print(w)

This will give you the size of the element. You want to click at the bottom of it, so you'd thing that you can just take the height, and pass that into move_by_offset like this: ".move_by_offset(0,h)". You can't do that, because when you select an element, it starts from the middle, so you want to cut that number in half (and round it down so that you don't have a decimal.) This is what I ended up doing that worked:

# import ActionChains
from selenium.webdriver.common.action_chains import ActionChains
import math

scroll_bar_xpath = "//div[@ng-if='::vm.isVirtual']/div[@class='ps-scrollbar-y-rail']"
element = driver.find_element(By.XPATH, scroll_bar_xpath)
size = element.size
w = size['width'] 
h = size['height']

#Calculate where to click
click_place = math.floor(h / 2)

# Do Stuff
ActionChains(driver).move_to_element(element).move_by_offset(0, click_place).click().perform() #50 worked

Hope it helps!