CODEWITHSUNDEEP
pythonweb-scraping
lumpy
lumpy

Reputation: 3

Getting item price with Web Scraping

I would like to make a Python script, but unfortunately, when I want to check the price, I get NONE instead of the price itself (or US$00.00 if I change the code). I found a lot of examples for it, where the HTML is with id, but I do not know how I should do it with class.

What I have so far:

import requests
from bs4 import BeautifulSoup


URL = 'https://www.banggood.com/BlitzWolf-BW-LT32-2M-USB-RGB-TV-Strip-Light-Kit-Sync-with-TV-Screen-Color-3-Sides-Cover-for-TV-Vivid-RGB-Color-Lighting-Effect-and-Dual-Simple-Control-p-1785284.html?akmClientCountry=HU&channel=facebookads&utm_source=facebookads&utm_medium=cpc&utm_campaign=app-dpa-all-ww-sm-afpur180-0407_prdshare_msg&utm_content=suki&utm_design=-&fbclid=IwAR3cAovrze4opttuwwpUxnXZ6dkRjhau3RqmDqASPW0mSEOI8sJgn9Cqss8&_branch_match_id=850392564428961100&cur_warehouse=CN&ID=45764'

headers = {"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537'}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')

    title = soup.find("span",{"class":'product-title-text'}).get_text()
    price = soup.find("span",{"class":'main-price'}).get_text()


converted_price = price[0:5]

print(converted_price)
print(title)

Upvotes: 0

Views: 3369

Answers (1)

Ananth
Ananth

Reputation: 831

BeautifulSoup has very minimal use to web scraping when the website is using Javascript and changes dynamically. Most of the websites these days you Javascript making it difficult to scrape data. One of the alternate option is to use Selenium.

If you have already used Selenium then directly jump to the code block below. If not, follow the instructions below.

  1. Check the Chrome version you are using in About Chrome under the options menu(top right corner of the browser).
  2. Go to this website and download the same version of the driver.
  3. Create a folder C:\webdrivers and copy the downloaded driver into this folder.
  4. Copy the file path C:\webdrivers\chromedriver.exe and add it to PATH in the environment variables (

Now execute the code below :

from selenium import webdriver

opts = webdriver.ChromeOptions()
# The below line will make your browser run in background when uncommented
# opts.add_argument('--headless')
    
driver = webdriver.Chrome(options= opts, executable_path = 'C:\webdrivers\chromedriver.exe')

driver.get("https://www.banggood.in/BlitzWolf-BW-LT32-2M-USB-RGB-TV-Strip-Light-Kit-Sync-with-TV-Screen-Color-3-Sides-Cover-for-TV-Vivid-RGB-Color-Lighting-Effect-and-Dual-Simple-Control-p-1785284.html?akmClientCountry=HU&channel=facebookads&utm_source=facebookads&utm_medium=cpc&utm_campaign=app-dpa-all-ww-sm-afpur180-0407_prdshare_msg&utm_content=suki&utm_design=-&_branch_match_id=850392564428961100&cur_warehouse=CN&ID=45764")

prices = driver.find_elements_by_class_name('main-price')

for price in prices:
    print(price.text)
driver.quit()

Output :

₹1,712.63

For more details about Selenium go through the documentation, it's a quite impressive module!

Upvotes: 1

Related Questions