CODEWITHSUNDEEP
pythonseleniumweb-scraping
Harsha
Harsha

Reputation: 1

Scrape allAmazon product names using selenium and python python

I am trying to scrape product names from https://www.amazon.com/Best-Sellers-Electronics-Televisions/zgbs/electronics/172659 this url, using selenium and python . I used css selectors and xpath, Although there are 50 products in the page around 30 product names are getting scraped after that I'm receiving the error,no element found. There's no issue with xpath.

I have tried using sleep before finding the element. It didn't work.

Please tell me know how to scrape the title for all 50 products. Thanks in advance.

Tried using sleep and also used scroll into view, windows.scrollto document.body . height. Nothing worked

Upvotes: 0

Views: 77

Answers (1)

Gowthaman Ravindran
Gowthaman Ravindran

Reputation: 65

I think its because the initial load has only the 30 products. The other 20 are loaded only when the page is scrolled to the bottom.

driver.execute_script("window.scrollTo(0,Math.max(document.documentElement.scrollHeight," + "document.body.scrollHeight,document.documentElement.clientHeight));");
items = driver.find_elements(By.XPATH, "//div[@id='gridItemRoot']//a/span/div")

This picked 50 items for me.

Upvotes: 1

Related Questions