CODEWITHSUNDEEP
pythonseleniumiframeselenium-webdriverwebdriver
Makiyo
Makiyo

Reputation: 451

Python: How to scrape data inside an iframe when you need to scroll down

enter image description here

I used webdriver to login to a website, and I found that the data I wanted to extract is inside an iframe. Like a picture above. At first, I used xpath helper to get me the whole data table.

select_box2 =driver.find_element_by_xpath("//table[@class='k-selectable']/tbody").text
print(select_box2)

But I can only extract 7 datas (Total are 400). Everytime when I scrolled down manually, I can see another part of data, but cannot do the whole at the same time. Any suggestion?

Upvotes: 0

Views: 1132

Answers (1)

Guilherme
Guilherme

Reputation: 1835

Recently i did a script to scrape data from instagram

there i need to sroll down the page to load more items

The strategy was:

  • get a refence of a element at the botton of the page;
  • scroll the page until the element be visible;

something like

driver.get(url)

items = driver.find_elements_by_tag_name("li") #get a list of items

q = len(items)
li_text = items[q-1].text.splitlines() #in this case i looking for the penultimate element
li = driver.find_element_by_xpath("//li[contains(.,'"+li_text[1]+"')]") #get the element reference with find_element()

li.location_once_scrolled_into_view #scroll down

source

Upvotes: 1

Related Questions