CODEWITHSUNDEEP
pythonseleniumselenium-webdriverwebdriverbeautifulsoup
plain vanilla
plain vanilla

Reputation: 45

Use webdriver,python,beautifulsoup to retrieve dynamic website

Trying to change the sort form the default (most relevant) into the "newest", something wrong and I can't save the page. Any suggestions?

#start webdriver to open the given product page via chrome browser
driver =webdriver.Chrome()
driver.get('http://www.homedepot.com/p/Husky-41-in-16-Drawer-Tool-Chest-and-Cabinet-Set-HOTC4016B1QES/205080371')
time.sleep(2)


#find the drop down list, then select the newest option and click  
m=driver.find_element_by_id("BVRRDisplayContentSelectBVFrameID")
m.find_element_by_xpath("//option[@value='http://homedepot.ugc.bazaarvoice.com/1999m/205080371/reviews.djs?format=embeddedhtml&sort=submissionTime']").click()
time.sleep(2)

#save the search result into the python
html = driver.page_source
file_object = open("samplereview.txt", "a")
file_object.write(str(html))
file_object.close( )
time.sleep(2)

soup=BeautifulSoup(html)

#quit from driver
driver.quit

Upvotes: 1

Views: 1025

Answers (1)

alecxe
alecxe

Reputation: 473803

You are missing two key selenium-specific things:

  • do not use time.sleep() - use Waits
  • use Select class when dealing with select/option - it provides a very nice abstraction

Here is the modified code (look how readable and short it is):

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select

driver = webdriver.Chrome()
driver.get('http://www.homedepot.com/p/Husky-41-in-16-Drawer-Tool-Chest-and-Cabinet-Set-HOTC4016B1QES/205080371')

# waiting until reviews are loaded
element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, 'BVRRDisplayContentSelectBVFrameID'))
)

select = Select(element)
select.select_by_visible_text('Newest')

And now I see the reviews sorted from newest to oldest:

enter image description here

To parse the reviews you don't necessarily need to pass the page source to the BeautifulSoup for further processing - selenium itself is powerful in locating elements:

reviews = []
for review in driver.find_elements_by_xpath('//span[@itemprop="review"]'):
    name = review.find_element_by_xpath('.//span[@itemprop="name"]').text.strip()
    stars = review.find_element_by_xpath('.//span[@itemprop="ratingValue"]').text.strip()
    description = review.find_element_by_xpath('.//div[@itemprop="description"]').text.strip()

    reviews.append({
        'name': name,
        'stars': stars,
        'description': description
    })

print(reviews)

Prints:

[
    {'description': u'Very durable product. Worth the money. My husband loves it',
     'name': u'Excellent product',
     'stars': u'5.0'},

    {'description': u'I now have all my tools in one well organized box instead of several boxes and have a handy charging station for cordless tools on the top . Money well spent. Solid box!',
     'name': u'Great!',
     'stars': u'5.0'},

    ...
]

Upvotes: 1

Related Questions