CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
josef_joestarr
josef_joestarr

Reputation: 153

Python, BS and Selenium

I try to webscrape with javascript dynamic + bs + python and Ive read a lot of things to come up with this code where I try to scrape a price rendered with javascript on a famous website for example:

from bs4 import BeautifulSoup
from selenium import webdriver

url = "https://www.nespresso.com/fr/fr/order/capsules/original/"

browser = webdriver.PhantomJS(executable_path = "C:/phantomjs-2.1.1-windows/bin/phantomjs.exe")
browser.get(url)
html = browser.page_source

soup = BeautifulSoup(html, 'lxml')

soup.find("span", {'class':'ProductListElement__price'}).text

But I only have as a result '\xa0' which is the source value, not the javascript value and I don't know really what I did wrong ...

Best regards

Upvotes: 1

Views: 207

Answers (2)

QHarr
QHarr

Reputation: 84475

You don't need the expense of a browser. The info is in a script tag so you can regex that out and handle with json library

import requests, re, json

r = requests.get('https://www.nespresso.com/fr/fr/order/capsules/original/')
p = re.compile(r'window\.ui\.push\((.*ProductList.*)\)')
data = json.loads(p.findall(r.text)[0])
products = {product['name']:product['price'] for product in data['configuration']['eCommerceData']['products']}
print(products)

Regex:

enter image description here

Upvotes: 1

Marsilinou Zaky
Marsilinou Zaky

Reputation: 1047

Here are two ways to get the prices 

from bs4 import BeautifulSoup
from selenium import webdriver

url = "https://www.nespresso.com/fr/fr/order/capsules/original/"

browser = webdriver.Chrome()
browser.get(url)
html = browser.page_source

# Getting the prices using bs4
soup = BeautifulSoup(html, 'lxml')
prices = soup.select('.ProductListElement__price')
print([p.text for p in prices])

# Getting the prices using selenium 
prices =browser.find_elements_by_class_name("ProductListElement__price")
print([p.text for p in prices])

Upvotes: 0

Related Questions