CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
David Clarke
David Clarke

Reputation: 55

BeautifulSoup not finding tags 3

BeautifulSoup is not finding the div tag 'pl-price js-pl-price'. I see it in the inspect element, however when I run my code, my code returns 'None'. Div 'product-details' is also in the HTML and it is found. But, div tag 'pl-price js-pl-price' can't be found by beautifulsoup. Why is that?

My Code:

import urllib2, sys, requests
from bs4 import BeautifulSoup

site = "https://www.lowes.com/pl/Refrigerators-Appliances/4294857973?goToProdList=true"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page, 'html.parser')

details = soup.find('div', attrs={'class': 'product-details'})
name = details.find('p', attrs={'class': 'h6 product-title js-product-title met-product-title v-spacing-large ellipsis-three-line art-plp-itemDescription'}) 
price = soup.find('div', attrs={'class': "product-pricing"})
actual_price = price.find('div', attrs={'class': "pl-price js-pl-price"})

print actual_price

HTML FROM WEBSITE:

<div class="product-details">
 <div class="product-pricing">
  <div class="pl-price js-pl-price" tabindex="-1">
  <!-- Start of Product Family Pricing -->
  <!-- Map price and savings through date present for product family -->

RESULTS:

scrape_products.py None

Upvotes: 0

Views: 155

Answers (1)

Steven
Steven

Reputation: 854

If you look at the results of the price search, you can see in the HTML the following text:

"Since Lowes.com is national in scope, we check inventory at your local store first in an effort to fulfill your order more quickly. You may find product or pricing that differ from that of your local store, but we make every effort to minimize those differences so you can get exactly what you want at the best possible price."

This would lead me to believe that the pricing information isn't loading immediately, and thus isn't loaded into the HTML that is parsed by BeautifulSoup. You should try a headless browser solution with Selenium.

Upvotes: 1

Related Questions