CODEWITHSUNDEEP
javascriptpythonseleniumweb-scrapinglinkedin-api
G. Macia
G. Macia

Reputation: 1521

Selenium Python: How to Scroll Down in a Pop Up window

I am working on a Linkedin web scraping project. I am trying to get the list of companies that interest someone (notice I am not using the API). It is a dynamic website, so I would need to scroll down while scraping the names of the companies. I know how to do this in the MAIN window, but since Interest are a pop-up window this solution to scroll does not work. My code so far was:

from selenium.webdriver.common.keys import Keys
bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
bar.send_keys(Keys.END)

Since it didn't work, I also tried:

bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', bar)

The problem is that I am not acting on the pop up window but in the main one so it has not the desired effect.

enter image description here

Upvotes: 8

Views: 29248

Answers (3)

PaulMuadDib
PaulMuadDib

Reputation: 61

I know this question is quite a few years old at this point. It does still come up in search on Google. I tried similar methods before similar to the answers in this thread. In my experience using Selenium, the JavaScript scroll method is great for getting dynamic content to load into a page. But it can be janky at times when trying to get specific elements to scroll into view. That could be from me not using JavaScript that often.

I ended up referring to the documentation Selenium provides and it looks like there is an element property that I didn't know about called location_once_scrolled_into_view which scrolls elements into view. It even works on those overlay popup windows. https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webelement.html#selenium.webdriver.remote.webelement.WebElement.location_once_scrolled_into_view

Upvotes: 1

Dhifaoui Bilel
Dhifaoui Bilel

Reputation: 51

first I tried with this script but it doesn't work:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    scrollable_popup.send_keys(Keys.END)
    time.sleep(2)

and then I used this script and it works fine with me:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_popup)
    time.sleep(2)

Upvotes: 2

Andersson
Andersson

Reputation: 52665

You can try to find element inside popup (the one that can be focused), for example some anchor:

element_inside_popup = driver.find_element_by_xpath('//div[@class="entity-list-wrapper ember-view"]//a')

and then use below code to scroll popup down:

from selenium.webdriver.common.keys import Keys

element_inside_popup.send_keys(Keys.END)

Upvotes: 9

Related Questions