CODEWITHSUNDEEP
pythonseleniumxpathweb-scraping
user1941390
user1941390

Reputation: 520

How to access grand childs of a node using XPath in Python Selenium?

I am preparing a web-scraping script which supposed to find list of attorneys in an area through a business directory web-site. I use chrome driver to fill out search keywords and the area values.

Since some of the hits don't have phone number, I would like to iterate through list of DIVs corresponding to search results and then check if the it has the phone number as a grand grand child and if yes then I get the phone number too otherwise I leave that field blank.

I have come up with two ways to do it as per below code.

import time
import json as js
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

url = 'http://www.yellowpages.com/search?search_terms=Divorce+Attorneys&geo_location_terms=sun+diego'
RsultsList = []

driver = webdriver.Chrome()
driver.get(url)
ThereIsNext = True
while ThereIsNext:
    
    # find ads.
    nAddResults = len( driver.find_elements_by_xpath("//div[@class='result flash-ad']"))

    #print 'add size = %d' % nAddResults
    for i in range(nAddResults):
        phone1 = driver.find_elements_by_xpath("//div[@class='result flash-ad']/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]")[i].text
        BusinessName1 = driver.find_elements_by_xpath("//div[@class='result flash-ad']//a[@class='business-name']")[i].text

        elem = driver.find_elements_by_xpath("//div[@class='result flash-ad']")
        phone2 = elem.find_element_by_xpath("/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]")[i].text
        BusinessName2 = elem.find_element_by_xpath("//a[@class='business-name']")

The first one is prone to error as records with no phone no. do NOT necessary show up at the end. So I come up with the second way. However, I get the below error message if try the second.

Traceback (most recent call last):
  File "C:\Users\XXXX\documents\visual studio 2015\Projects\PythonApplication3\
PythonApplication3\AtorneyList.py", line 23, in <module>
    phone2 = elem.find_element_by_xpath("/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1
]").text
AttributeError: 'list' object has no attribute 'find_element_by_xpath'
Press any key to continue . . .

Please let me know what I am missing. I have checked this and that already and couldn't understand.

Much appreciate it.

Thanks

Upvotes: 0

Views: 746

Answers (1)

Saurabh Gaur
Saurabh Gaur

Reputation: 23815

Actually find_elements() returns either list of WebElement or empty list. You're storing this result into a list variable name elem.

AttributeError: 'list' object has no attribute 'find_element_by_xpath'

This occurs because you're going to find nested WebElement on elem list that's why you're calling as elem.find_element_by_xpath() which is absolutely wrong.

Actually find_element() or find_elements is used to search the element on the page context or the context of the WebElement instead of list.

So you should try to find list of WebElement from driver means page context and then iterate to find further nested WebElement using this element context as below :-

elems = driver.find_elements_by_xpath("//div[@class='result flash-ad']")

for elem in elems:
    phone2 = elem.find_element_by_xpath(".//div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]").text
    BusinessName2 = elem.find_element_by_xpath(".//a[@class='business-name']").text

Upvotes: 1

Related Questions