Reputation: 2477
I am trying to fetch the tweets from a twitter url. But when I execute the code, the following error occurs some of the times :
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=84.0.4147.89)
The error points to this line :
---> 18 print(tweet.text)
This is the corresponding code :
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import csv
browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/'+hashtag+'?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')
tweet_contents = []
for _ in range(5):
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
for tweet in tweets:
print(tweet.text)
tweet_contents += [tweet.text]
body.send_keys(Keys.PAGE_DOWN)
time.sleep(0.4)
Any help or suggestions would be really appreciated.
Upvotes: 4
Views: 7857
Reputation: 214
seems like it takes the tweets to load more than 0.4 seconds
i copy pasted the code with changing the sleep time from 0.4 to 2 seconds and it worked fine.
browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/' + hashtag + '?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')
tweet_contents = []
for _ in range(5):
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
for tweet in tweets:
print(tweet.text)
tweet_contents += [tweet.text]
body.send_keys(Keys.PAGE_DOWN)
time.sleep(2)
if you see it works on your env consider increasing the sleep time from 0.4 to some higher number.
another option is to use a retry mechanism.
Upvotes: 3
Reputation: 2477
The web element tweets
found before the for loop becomes inaccessible inside the for loop at the end for some reason.
So I tried initiating tweets
again inside the for loop and it works now. It seems to be a temporary fix, but works.
The last block of my code now :
tweet_contents = []
for _ in range(5):
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
time.sleep(1)
for tweet in tweets:
print(tweet.text)
tweet_contents += [tweet.text]
tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # initialising tweets variable again inside for loop
body.send_keys(Keys.PAGE_DOWN)
time.sleep(0.4)
tweet_contents = sorted(set(tweet_contents), key=tweet_contents.index) # To remove duplicates, whilst preserving the order of tweets
Upvotes: 0
Reputation: 978
Stale element exception means Web element is there on page but driver instance could not interact with that element.
There are 2 ways to overcome this stale element issue
1.Using refresh method in selenium (driver.navigate().refresh();)
2.Using for loop try to looping until that element is click
Upvotes: 0