CODEWITHSUNDEEP
pythonseleniumweb-scrapingtwitterweb-crawler
padul
padul

Reputation: 174

selenium - get all tweets after scrolling the website - python

my problem is actually two. One is that I scroll down until it doesn't work anymore and then try to save all answers. Unfortunately, I only get a small part of the lower answers. Is there a way to get all answers? I try it with a sleep time, but it does not work.

My second problem is that on some pages a button appears at the bottom of the page to click to get more answers. But I have not found a way to click on it yet.

I would be really grateful for a tip


url = 'https://twitter.com/RegSprecher/status/1251100551183507456'

driver = webdriver.Chrome(r"path_chromedriver.exe")
driver.implicitly_wait(10)
driver.get(url)


# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(1)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        # If heights are the same it will exit the function
        break
    last_height = new_height

#Wait
time.sleep(30)

#tweet id

tweet_id = driver.find_elements_by_css_selector("a[href*='status']")

for tweet in tweet:
    print(tweet.text)

for tweet_id in tweet_id:
    print(tweet_id.get_attribute('href'))


driver.quit()


https://twitter.com/MarkFin79124805/status/1251129277787131904
https://twitter.com/Ehrenfrau3/status/1251272923668787200
https://twitter.com/K30107265/status/1251788504318828549
https://twitter.com/Sakasonis/status/1251102005910818817
https://twitter.com/MattCone3/status/1251117184534949888
https://twitter.com/Volksdichter/status/1251186371160682502
https://twitter.com/Volksdichter/status/1251186371160682502/photo/1
https://twitter.com/RiaIssa/status/1251817059517947910
https://twitter.com/janejane24/status/1251102104736989184
https://twitter.com/RiaIssa/status/1251102636071403522
https://twitter.com/TiBo01774121/status/1251108273241104384
https://twitter.com/RiaIssa/status/1251195169937993733

Code from the Button

<div class="css-1dbjc4n r-my5ep6 r-qklmqi r-1adg3ll">
<div aria-haspopup="false" role="button" data-focusable="true" tabindex="0" class="css-18t94o4 css-1dbjc4n r-1777fci r-1jayybb r-o7ynqc r-6416eg r-13qz1uu">
<div class="css-1dbjc4n r-16y2uox r-1wbh5a2 r-1777fci">
<div dir="auto" class="css-901oao r-1n1174f r-1qd0xha r-a023e6 r-16dba41 r-ad9z0x r-bcqeeo r-q4m81j r-qvutc0"><span class="css-901oao css-16my406 r-1qd0xha r-ad9z0x r-bcqeeo r-qvutc0">Weitere Antworten anzeigen</span></div></div></div></div>

Upvotes: 0

Views: 1515

Answers (1)

SeleniumUser
SeleniumUser

Reputation: 4177

Try below solution :: 

url = 'https://twitter.com/RegSprecher/status/1251100551183507456'
driver.get(url)
driver.maximize_window()
wait = WebDriverWait(driver, 20)

scrolls = 7
while True:
    scrolls -= 1
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
    time.sleep(3)
    if scrolls < 0:
        break
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'Weitere Antworten anzeigen')]"))).click()

Note : please add below imports to your solution

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

Upvotes: 3

Related Questions