CODEWITHSUNDEEP
pythonseleniumweb-scraping
Abhishek Rai
Abhishek Rai

Reputation: 2237

Cannot locate child element Selenium

Trying to pick out single restaurant elements under the All Restaurants category on this page. https://www.foodpanda.sg/restaurants/new?lat=1.2915902&lng=103.8379066&vertical=restaurants

Element

All li elements have a different class name. The only working xpath, I have been able to figure out is this one.

//ul[@class="vendor-list"]//li[3]

I cannot figure out how to increase this number and get all the restaurants, which is complicated by the fact that there is an infinite scroll as well.

j = 1
def get_rest():
    global j
    while True:
        driver.execute_script("window.scrollBy(0,2525)", "")
        time.sleep(5)
        var = f'{[j]}'
        elems = driver.find_elements_by_xpath(f'//ul[@class="vendor-list"]//li{var})
        return elems

Upvotes: 0

Views: 66

Answers (2)

cruisepandey
cruisepandey

Reputation: 29382

I tried to get the innerHTML of the //ul[@class="vendor-list"]//li

Code :

driver.get("https://www.foodpanda.sg/restaurants/new?lat=1.2915902&lng=103.8379066&vertical=restaurants")
while True:
    for item in driver.find_elements(By.XPATH, "//ul[@class='vendor-list']/li"):
      ActionChains(driver).move_to_element(item).perform()
      sleep(0.1)
      print(item.get_attribute('innerHTML'))

Upvotes: 0

Prophet
Prophet

Reputation: 33361

I guess it should be something like this

all_restaurants = set()
restaurant_locator = '//ul[@class="vendor-list"]//li[@data-testid and not(@class)]'
page_bottom_locator = '.restaurants__city-bottom-info'
while #bottom is not reached:
    restaurants = driver.find_elements_by_xpath(restaurant_locator)
    all_restaurants.add(restaurants)
    time.sleep(1)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Upvotes: 1

Related Questions