CODEWITHSUNDEEP
pythonselenium
Drij Vyas
Drij Vyas

Reputation: 23

Python Selenium driver.get() not opening all of my URLs

i have a piece of code as follows (not the full code, just the relevent snippet for my query): 


linkedin_urls = driver.find_elements_by_xpath('//*[@href]')

for linkedin_url in linkedin_urls:
   hopeful = linkedin_url.get_attribute('ping')
   if hopeful and len(hopeful) > 0:
       actual = hopeful[31:]
       new = actual[:-50]
       if new.startswith('https://uk.linkedin.com'):
          print(new) 
          #driver.get(new)
          time.sleep(2)

When i print the results, each linkedin URL correctly prints without error. When i manually copy and paste these into a search bar, they each open the desired profiles. However, if i remove the # in front of driver.get(new) and run that part, it opens the first linkedin url in the list, and then errors with the following 

"Message: stale element reference: element is not attached to the page document".

I am not sure why as all the urls print correctly, and each one opens the desired page if they are manually copied and pasted into a search bar. i would like all the urls to open one by one.

Any help will be much appreciated!

Kind regards,

Drij

Upvotes: 2

Views: 114

Answers (3)

KunduK
KunduK

Reputation: 33384

"Message: stale element reference: element is not attached to the page document".

This explains that when you navigate to the new page the page get refreshed and initialized elements no longer reference to that page.

To resolved this problem you need to re-initialized the elements again.

linkedin_urls = driver.find_elements_by_xpath('//*[@href]')

for linkedin_url in range(len(linkedin_urls)):

   #Re-Initialized the elements again 
   linkedin_urls = driver.find_elements_by_xpath('//*[@href]')

   hopeful =linkedin_urls[linkedin_url].get_attribute('ping')
   if hopeful and len(hopeful) > 0:
       actual = hopeful[31:]
       new = actual[:-50]
       if new.startswith('https://uk.linkedin.com'):
          print(new)
          driver.get(new)
          driver.back()
          time.sleep(2)

Hope you are looking after this.

Upvotes: 1

Hamza Khurshid
Hamza Khurshid

Reputation: 825

As ZoulouDu explained, the error is due to the reference to the DOM elements of previous page from where you scraped the URLs. Now when a new page is loaded, the old reference is lost which causes the error.

What you can do is to convert these referenced strings to actual strings. Just use the following line for creating your new list instead of the one you are using.

new=[str(b) for b in actual[:-50]]

Upvotes: 0

MatthiasDec
MatthiasDec

Reputation: 175

When the line driver.get(new), the elements your a finding with your driver.find_elements_by_xpath('//*[@href]') are being removed from your DOM, this explains your error message.

Upvotes: 1

Related Questions