CODEWITHSUNDEEP
pythonseleniumweb-scraping
user4069366
user4069366

Reputation:

unable to iterate over elements in selenium python

Im a selenium noob and have been struggling to get things done with python. Im trying to iterate over all user reviews("partial_entry" class) from this page https://www.tripadvisor.com/Airline_Review-d8729164-Reviews-Cheap-Flights-or560-TAP-Portugal#REVIEWS

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome("C:\Users\shalini\Downloads\chromedriver_win32\chromedriver.exe")
driver.maximize_window()
url="https://www.tripadvisor.com/Airline_Review-d8729164-Reviews-Cheap-Flights-or560-TAP-Portugal#REVIEWS"
driver.get(url)

for i in driver.find_elements_by_xpath("//div[@class='wrap']"):
    print i.find_element(By.XPATH, '//p[@class="partial_entry"]')
        print i.text
    print "=============================================="
    # THIS IF BLOCK IS NECESSARY, I CANT DO AWAY WITH THIS ONE
    if i.find_elements(By.CSS_SELECTOR,"#REVIEWS .googleTranslation>.link"):
        print "======YES TRANSLATION AVAILABLE========"

Even though Im selecting a different element each time in for loop, but its printing the same element over and over again. (I have to keep the last if block & cant do away with it, so whatever the solution be it has to include that if block also)

======EDIT===================

Even this isnt working (which should actually work, according to http://selenium-python.readthedocs.io/locating-elements.html ) . I dont know whats going on with selenium !!!!!

print i.find_element(By.CSS_SELECTOR, 'p.partial_entry')

Output:

NoSuchElementException:

Upvotes: 1

Views: 2718

Answers (1)

aneroid
aneroid

Reputation: 15962

1. The reason you keep getting the first element repeatedly while iterating over i.find_element(By.XPATH, '//p[@class="partial_entry"]') in the 2nd loop is that the beginning // tries to locate elements from the root/top level, not as a descendant element of i. So that just keeps returning the first p.partial_entry element for each iteration of the outer loop.

To search for descendant elements of i which match p[@class="partial_entry"], the xpath should start with .//. That's what the dot does.

2. For the line print i.find_element(By.CSS_SELECTOR, 'p.partial_entry'):
The single find_element either return the first found element or throw an error if none are found. There are some 'div.wrap's which don't have that descendant element, so you will get NoSuchElementException.

The find_elements (note the 's') methods return a list of elements or an empty list if none are found, rather than an error.

So putting all that together:

>>> for i in driver.find_elements_by_xpath("//div[@class='wrap']"):
...     for ent in i.find_elements_by_xpath('.//p[@class="partial_entry"]'):
...         print ent.text
...         if i.find_elements_by_css_selector('#REVIEWS .googleTranslation>.link'):
...             print 'translation available'
...     print  # output clarity
...

Btw, why are you mixing things like find_elements_by_xpath('...') along with find_element(By.XPATH, '...')? Stick to one pattern.

Upvotes: 7

Related Questions