CODEWITHSUNDEEP
pythonseleniumweb-crawler
learningCode
learningCode

Reputation: 3

web crawling a table of links

I'm creating a script in python that goes through a table with three columns. I created a list where every link in the first column is inserted into the list. And then I loop through. When looping, I click into the link, print a statement to make sure it actually clicked into the link, and then go to the previous page so that the next link can be clicked. The error I keep getting is that my loop goes through the first two links first and then I get a StaleElementReferenceException when the loop calls links[page].click() for the third time. I can't post the html because the site is confidential. 

    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import Select
    import traceback


    # starting chrome browser
    chrome_path = r"C:\Users\guaddavi\Downloads\chromedriver_win32    extract\chromedriver.exe"
    browser = webdriver.Chrome(chrome_path)


    #linking to page
    browser.get('link to page with table ')


    #find table of ETL Extracts
    table_id = browser.find_element_by_id('sortable_table_id_0')
    #print('found table')

    #get all the rows of the table containing the links
    rows = table_id.find_elements_by_tag_name('tr')

    #remove the first row that has the header
    del rows[0]
    current = 0
    links = [] * len(rows)

    for row in rows:
     col = row.find_elements_by_tag_name('td')[0]
     links.append(col)
     current +=1

    page = 0
    while(page <= len(rows)):
        links[page].click()
        print('clicked link' + "  " + str(page))
        page += 1
        browser.back()

Upvotes: 0

Views: 316

Answers (1)

narko
narko

Reputation: 3875

I am not sure you already saw the official Selenium documentation:

A stale element reference exception is thrown in one of two cases, the first being more common than the second: The element has been deleted entirely. The element is no longer attached to the DOM.

In your case I think your are having the second issue. Every time you click and go back in the loop your DOM is changing. Please check that out.

Upvotes: 1

Related Questions