CODEWITHSUNDEEP
vbaseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

How can I scroll down a webpage using Selenium with VBA

I've written a script using VBA in combination with selenium to get all the company links from a webpage which doesn't display all the links until scrolled downmost. However, when I run my script, I get only 20 links but there are 1000 links in total. I've heard that it is possible to accomplish this type of task executing javascript function between the code. At this point, I can't get any idea how can I place that within my script. Here is what I've tried so far:

Sub Testing_scroll()

Dim driver As New WebDriver
Dim posts As Object, post As Object

driver.Start "chrome", "http://fortune.com/fortune500"
driver.get "/list/"

driver.execute_script ("window.scrollTo(0, document.body.scrollHeight);") --It doesn't support here

Set posts = driver.FindElementsByXPath("//li[contains(concat(' ', @class, ' '), ' small-12 ')]")

For Each post In posts
    i = i + 1
    Cells(i, 1) = post.FindElementByXPath(".//a").Attribute("href")
Next post

End Sub

Upvotes: 0

Views: 9601

Answers (2)

Bill Kepner
Bill Kepner

Reputation: 11

This works for me. It scrolls incrementally down the page until it reaches the end. I found that scrolling directly to the end didn't load all of the in-between elements. This example is Selenium VBA for a Chrome driver.

Sub scrollToEndofPage(dr As WebDriver)
Dim aCH As WebActionChain, scrPOS As Long, oldPOS As Long
scrPOS = dr.ExecuteScript("return window.pageYOffset;")
Set aCH = dr.ActionChain
oldPOS = -1
Do While scrPOS > oldPOS
    oldPOS = scrPOS
    aCH.ScrollBy 0, 100: aCH.Perform
    scrPOS = dr.ExecuteScript("return window.pageYOffset;")
    DoEvents
Loop
End Sub

Upvotes: 1

NotInventedHere
NotInventedHere

Reputation: 312

According to the examples included with SeleniumBasic you should be using

driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);")

not "driver.execute_script", which is the python equivalent from the previous solution I gave you :) You are going to have to loop that in the same way until you've got all 1000 links on the page.

Upvotes: 6

Related Questions