CODEWITHSUNDEEP
pythonselenium
bandcar
bandcar

Reputation: 743

find element by xpath not defined

When I run

# change the value inside the range to save the number of reviews we're going to grab
for i in range(0, pages_to_scrape):

    # give the DOM time to load
    time.sleep(2) 

    # Click the "expand review" link to reveal the entire review.
    find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

    # Now we'll ask Selenium to look for elements in the page and save them to a variable. First lets define a container that will hold all the reviews on the page. In a moment we'll parse these and save them:
    container = driver.findElement(By.xpath("//div[@data-reviewid]"))

I get the error

NameError                                 Traceback (most recent call last)
/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_18848/426937913.py in <module>
      6 
      7     # Click the "expand review" link to reveal the entire review.
----> 8     find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()
      9 
     10     # Now we'll ask Selenium to look for elements in the page and save them to a variable. First lets define a container that will hold all the reviews on the page. In a moment we'll parse these and save them:

NameError: name 'find_element_by_xpath' is not defined

I used print(dir(driver)) to see what command i'm supposed to use, and the one I am using is the correct one. I was originally using driver.findElement(By.xpath(".//div[contains(@data-test-target, 'expand-review')]").click(), which worked, but when combined with other codes gave the same error above, except it said "By" was not defined. Then I tried a user suggestion driver.find_element(By.XPATH, 'your xpath'), which also generated the same error.

Full code:

import selenium
import csv #This package lets us save data to a csv file
from selenium import webdriver #The Selenium package we'll need
import time #This package lets us pause execution for a bit

path_to_file = "/Users/user/Desktop/HotelReviews.csv"

pages_to_scrape = 3

url = "https://www.tripadvisor.com/Hotel_Review-g60982-d209422-Reviews-Hilton_Waikiki_Beach-Honolulu_Oahu_Hawaii.html"

# import the webdriver
driver = webdriver.Safari()
driver.get(url)

# open the file to save the review
csvFile = open(path_to_file, 'a', encoding="utf-8")
csvWriter = csv.writer(csvFile)

# change the value inside the range to save the number of reviews we're going to grab
from selenium.webdriver.common.by import By
for i in range(0, pages_to_scrape):

    # give the DOM time to load
    time.sleep(2) 

    # Click the "expand review" link to reveal the entire review.
    driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()

  
# When all pages have been processed, quit the driver
driver.quit()

EDIT: Here is the updated code with new errors

from selenium.webdriver.common.by import By
for i in range(0, pages_to_scrape):

# give the DOM time to load
time.sleep(2) 

# Click the "expand review" link to reveal the entire review.
driver = webdriver.Safari() #url was defined in a different portion of the code
driver.get(url)
driver.find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

which generates

  File "/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_24069/1057492970.py", line 15
    driver.find_elements(by=By.XPATH,".//div[contains(@data-test-target, 'expand-review')]").click()
                                                                                           ^
SyntaxError: positional argument follows keyword argument

Following the format in the error code, I ran

driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()

Which then gave another error:

File "/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_24069/4123117837.py", line 11
    driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()
                                                                                            ^
SyntaxError: positional argument follows keyword argument

Upvotes: 1

Views: 4833

Answers (2)

si ho
si ho

Reputation: 1

Use this line instead:

from selenium.webdriver.common.by import By

Upvotes: 0

JimShapedCoding
JimShapedCoding

Reputation: 917

You are calling find_element_by_xpath directly.

All the selenium methods need to be called from the driver object after you instantiate it, declaring the website you are scraping.

So the steps are:

  • Create the driver instance
  • Use get() to go to the website you want to scrape
  • Find your elements by XPath. But, refer to the methods from the driver instance

Example:

driver = webdriver.Chrome("path/to/driver")
driver.get("yourwebsite.com")
driver.find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

Upvotes: 1

Related Questions