CODEWITHSUNDEEP
pythonpython-3.xseleniumweb-scrapinginfinite-scroll
babban babu
babban babu

Reputation: 93

Python selenium - How to access the inner scroll inside the main scroll

I am trying to scrape all the data in the website as shown in the image. here For that, I'll need to keep on scrolling down the infinite scroll. But I am unable to access the inner scroll. I use the following code for infinite scrolling

last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

It doesn't work. It scrolls down the main outer scroll instead of the inner one, as shown below. Kindly help me scroll down the inner scroll

enter image description here

Upvotes: 1

Views: 2944

Answers (1)

Max Daroshchanka
Max Daroshchanka

Reputation: 2978

1 You have to find the scrollable element.

Try some of this suggestions

Find first scrollable parent

2 And then

scrollable_element = driver.find_element... # locate using css or xpath the element, found during point 1 research

driver.execute_script("arguments[0].scroll(0, arguments[0].scrollHeight);", scrollable_element)

Upvotes: 4

Related Questions