CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingbeautifulsoup
Rwood90
Rwood90

Reputation: 19

BS4 - returning 'None' when requesting Span

I have (another) quick question on BS4 when I am trying to take a price from a website. A snippet of the html code is below:

        <div class="summary entry-summary col-md-7">
                            <h2 class="product_title entry-title">
        Tecsound SY50 Self Adhesive (7.26m2)    </h2>
<p class="price"><span class="woocommerce-Price-amount amount"><bdi><span class="woocommerce-Price-currencySymbol">£</span>61.50</bdi></span> <small class="woocommerce-price-suffix">(<span class="woocommerce-Price-amount amount"><bdi><span class="woocommerce-Price-currencySymbol">£</span>73.80</bdi></span> inc vat)</small></p>

The price I am trying to get to is the £61.50 (which is the full price without GB taxes).

my attempt is:

import requests
from bs4 import BeautifulSoup

url = requests.get("https://www.the-website.co.uk")
vari = url.content
soup = BeautifulSoup(vari, "html.parser")
price = soup.find('span', {'class': 'woocommerce-Price-amount amount'})
print(price)

Output:

None

... and I have tried a similar:

import requests
from bs4 import BeautifulSoup
    
url = requests.get("https://www.the-website.co.uk")
vari = url.content
soup = BeautifulSoup(vari, "html.parser")
span = soup.find('span', {'class': 'woocommerce-Price-amount amount'})
price = span.text
print(price)

this produces the error:

AttributeError: 'NoneType' object has no attribute 'text'

The second error is expected, as I am clearly not picking up the information with my request, so there is nothing or to show, or 'None' as shown in my first example.

Please can somebody help me to correctly pluck the information I require from the html text?

Upvotes: 1

Views: 513

Answers (1)

HedgeHog
HedgeHog

Reputation: 25073

As mentioned by Carcigenicate you should use selenium - You also could improve your question by providing the real url.

Example

from selenium import webdriver
from bs4 import BeautifulSoup
from time import sleep

url = "https://soundproofingstore.uk/product/tecsound-sy50-self-adhesive/"

driver = webdriver.Chrome(executable_path=r'C:\Program Files\ChromeDriver\chromedriver.exe')

driver.get(url)
sleep(3)
soup = BeautifulSoup(driver.page_source,"lxml")
price = soup.find('span', {'class': 'woocommerce-Price-amount amount'}).get_text()
print(price)

driver.close()

Output

£61.50

Upvotes: 1

Related Questions