CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverselenium-chromedriver
technophile_3
technophile_3

Reputation: 521

Why am I not able to display the product price using selenium in python?

I am scraping a website to get the product price using selenium. The program does work like it should ie. The intention is to Enter the product name(which is done programmatically)->after product is displayed-> it should display the price but I get an empty list. The price is not getting printed. Here's my code.

def scrape_parkCams(product_name):
    website_address = 'https://www.parkcameras.com'
    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)

    browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)

    browser.get(website_address)
    time.sleep(2)

    browser.find_element_by_id('searchbar').send_keys(product_name)
    browser.find_elements_by_tag_name('ff-searchbutton')
    # browser.find_elements_by_class_name('icon-search')[0].submit()

    page_source = browser.page_source

    soup = BeautifulSoup(page_source, 'lxml')
    product_price_raw_list = browser.find_elements_by_xpath('//strong[@class="priceContainer record-product-price"]')
    product_price_list = [elem.text for elem in product_price_raw_list]
    print(product_price_list)
    return product_price_list


if __name__ == "__main__":
    scrape_parkCams('Canon EOS 90D Digital SLR Body')

The output that I get is : []

The output that I should be getting is : ['£2,450.00','£3,450.00'] I am not sure if I have used the xpath in the correct form,is it correct? Please help.

Upvotes: 0

Views: 353

Answers (2)

Arundeep Chohan
Arundeep Chohan

Reputation: 9969

This is based of a what the site gives to me with all these popups and everything. You can use webdriver waits to get rid of the time.sleep(), you never clicked the search bar. I had a slightly different xpath.

browser.get('https://www.parkcameras.com')
time.sleep(10)
browser.find_element_by_xpath("//a[contains(@id,'popup-subcription-closes-icon')]").click()
browser.find_element_by_css_selector('a.cookie-banner__close').click()
browser.find_element_by_id('searchbar').send_keys(product_name)
browser.find_element_by_css_selector('ff-searchbox > ff-searchbutton > button').click()
time.sleep(5)
product_price_raw_list = browser.find_elements_by_xpath('//div[@class="priceContainer record-product-price"]/p')
product_price_list = [elem.text for elem in product_price_raw_list]
print(product_price_list)

Outputs

['£1,209.00', '£469.00', '£549.00', '£369.00', '£2,787.00', '£1,599.00', '£1,299.00', '£1,349.00', '£429.00', '£379.00', '£839.00', '£839.00']

Upvotes: 1

Bahrudin Trbalic
Bahrudin Trbalic

Reputation: 1

Most prices are not hard-coded in the HTML - they require a query to the database which takes time or has some protective measures. Open the web page inspector, inspect by element, navigate to the price box and then check the Network to see the times data arrives. You might want to delay your script by the amount the data takes.

Upvotes: 0

Related Questions