Trouble targeting elements on website (selenium webdriver)

Question

I am trying to target properties on a real estate website. Ideally, I want to pull the property marketing URL, the title, location, and email of each listing. The properties are all listed as so:


        

           
             
                 
                   
                
            
            
                

                    Engelberg Antik's

                
                
                    


                    Salem, OR

                
                

                    Outstanding downtown Salem opportunity, right next…

                
                
                    
                    
                    
                
            
            
                

                    $479,900

                
                

                    13,612 SF

                
                

                    Street Retail

                
            
        
    


    ::before
    


    or call
    503.588.0400
    for more information

...

this was my first attempt at it:

from selenium import webdriver
import sys
import smtplib
import pymongo

newProperties = []

driver = webdriver.Firefox()
driver.get('http://svncommercialadvisors.com/properties/')

for property in driver.find_elements_by_class_name('propertyList'):
    #get title,location 
    info = property.find_elements_by_class_name('addressInfo')
    email = property.find_elements_by_partial_link_text('.com')

When I run the above, it doesn't give any errors that the driver can't locate elements. However, when I print out the elements nothing appears. How can I better locate the elements? I would like for something like this, appended to a list:

-title: Engelberg Antik's
-location: Salem, OR
-url: http://svncommercialadvisors.com/properties/?propertyId=74495-sale
-email: brokeremail@svn.com

alecxe · Accepted Answer

The key problem here is that the search results are loaded in an iframe.

You need to switch to iframe before searching for properties.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get('http://svncommercialadvisors.com/properties/')

# wait for frame to appear and switch
frame = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div#buildout iframe")))
driver.switch_to.frame(frame)

for property in driver.find_elements_by_class_name('propertyList'):
    info = property.find_element_by_class_name('addressInfo')
    email = property.find_element_by_partial_link_text('Email')

    print info.text
    print print email.get_attribute('href')

I've also applied two fixes:

replaced find_elements_by_class_namme with find_elements_by_class_name
replaced property.find_elements_by_partial_link_text('.com') with property.find_element_by_partial_link_text('Email')

It prints:

Engelberg Antik's
Salem, OR
Outstanding downtown Salem opportunity, right next door to the newly renovated Roth and McGilchri...
mailto:jennifer.martin@svn.com

Trouble targeting elements on website (selenium webdriver)

Answers (1)

Related Questions