CODEWITHSUNDEEP
pythonselenium-webdrivertwitterweb-scraping
Rakib
Rakib

Reputation: 7625

Page scrolling using Selenium is not working

I need all the replies/comments of a tweet. The related question has an answer which requires to download too much data and then discard them after cross matching, and it is not possible for me due to the rate limitations. I tried to scrape the page by first loading the tweet url using python. To scroll the page, I tried to use selenium web driver. But I still get only replies in the first page. For some reason, scrolling is not working. I tried these 1,2,3, 4 approaches, but none worked in this case.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Firefox()
driver.get("https://twitter.com/neiltyson/status/912299342559694848")

for in xrange(10):
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    print('height:{}'.format(driver.execute_script("return document.body.scrollHeight")))
    time.sleep(3)

I noticed that, the height does not change after the first iteration.

Upvotes: 0

Views: 1851

Answers (1)

ivan7707
ivan7707

Reputation: 1156

I have Python3 running right now, so I changed xrange to range to test it out.

Try this (works for me):

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Firefox()
driver.get("https://twitter.com/neiltyson/status/912299342559694848")

page = driver.find_element_by_tag_name('body')

for i in range(10):
    page.send_keys(Keys.PAGE_DOWN)
    time.sleep(3)

Upvotes: 1

Related Questions