CODEWITHSUNDEEP
pythonseleniumweb-scraping
from_the_m00n
from_the_m00n

Reputation: 1

web scraper is not scrapping the complete page

I want to scrape and display the names of all the cars from this dealership webpage:

https://www.herbchambers.com/used-inventory/index.htm?geoZip=02108&geoRadius=0

I located the corresponding x-path and figured out the pattern within it, to find the x-paths of every single car name on the page.

x = 1
while True:
    the_xpath = f"/html/body/div[2]/div/div/div[8]/div/div[2]/div[1]/div/ul/li[{x}]/div[1]/div[2]/h2/a"
    car_name = driver.find_element(By.XPATH, the_xpath)
    car_name.location_once_scrolled_into_view
    print(car_name.text)
    x += 1

It works perfectly fine and prints the names of the first 7-9 cars (varies every time). However, it then always terminates with the NoSuchElementException, without finishing the entire page.

I was wondering if anyone could help me solve this issue and figure out why it only works half way.

Upvotes: 0

Views: 422

Answers (1)

Mohammed
Mohammed

Reputation: 11

generally your way of defining the path is risky when working with selenium as absence of any element inside the box that contains the car will make a mess and selenium will return that error for example if there's divisons and each division represent a car then we have 3 elements (generated by javascript if present) inside each first for name one for color and one for price if you want to collect the price by defining the 3rd element of each division if the color of a car not present and its element not generated then you will have a crash as now that car only has 2 elements

the best approach is to find attribute with values representing the information for example in your case you first find div by class 'vehicle-card-details-container' this div contains h2 then a like this

car_name = driver.find_element(By.XPATH, '//div[contains(@class,"vehicle-card-details-container")]//h2/a')
car_name.location_once_scrolled_into_view
print(car_name.text)
x += 1

Upvotes: 0

Related Questions