CODEWITHSUNDEEP
pythonpython-3.xseleniumweb-scraping
Suraj
Suraj

Reputation: 2477

**Python selenium error** :Message: stale element reference: element is not attached to the page document

I am trying to fetch the tweets from a twitter url. But when I execute the code, the following error occurs some of the times :

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=84.0.4147.89)

The error points to this line :

---> 18         print(tweet.text)

This is the corresponding code :

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import csv

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/'+hashtag+'?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
    
    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.4)

Any help or suggestions would be really appreciated.

Upvotes: 4

Views: 7857

Answers (3)

yogev
yogev

Reputation: 214

seems like it takes the tweets to load more than 0.4 seconds

i copy pasted the code with changing the sleep time from 0.4 to 2 seconds and it worked fine.

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/' + hashtag + '?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")

    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(2)

if you see it works on your env consider increasing the sleep time from 0.4 to some higher number.

another option is to use a retry mechanism.

Upvotes: 3

Suraj
Suraj

Reputation: 2477

The web element tweets found before the for loop becomes inaccessible inside the for loop at the end for some reason.

So I tried initiating tweets again inside the for loop and it works now. It seems to be a temporary fix, but works.

The last block of my code now :

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
    time.sleep(1)
    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
        tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # initialising tweets variable again inside for loop
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.4)

tweet_contents = sorted(set(tweet_contents), key=tweet_contents.index) # To remove duplicates, whilst preserving the order of tweets

Upvotes: 0

Justin Lambert
Justin Lambert

Reputation: 978

Stale element exception means Web element is there on page but driver instance could not interact with that element.

There  are 2 ways to overcome this stale element issue 
1.Using refresh method in selenium (driver.navigate().refresh();)
2.Using for loop try to looping until that element is click

Upvotes: 0

Related Questions