CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
Chaki0981
Chaki0981

Reputation: 11

Beautiful Soup find function returns None

for some reason find function return none, even though it exists in html code. First find: price = row.find('div', {'class': 'price-normal selenium-price-normal'}) return normally the price, but second one: price = row.find('div', {'class': 'price-normal'}) Return none and when i try to print text out of it I get error even though HTML looks very similar: AttributeError: 'NoneType' object has no attribute 'text'

Here is HTML, from where I can not extract data:

<div class="product-price  price-show instalments-show">
                <div class="price-normal">
                    4&nbsp;095&nbsp;zł</div>
                <div class="price-instalments-box">
                        <div class="price-instalments-row">
                            <span class="price-instalments-label">
                                    RATY:</span>
                            <a href="/telefony-komorkowe/apple-iphone-12-256gb-zielony.bhtml#outlet-doskonaly|raty" class="price-instalments selenium-price-instalments js-scroll-by-hash">
                                682,50 zł <span>x 6 rat 0%</span><i></i>
                                </a>
                        </div>
                    </div>
                </div>

Here is HTML that works:

<div class="product-prices-box">
        <div class="product-price product-price-101611036529 price-show instalments-show">
    <div class="price-presentation">
                    <div class="price-normal selenium-price-normal">
    5&nbsp;499&nbsp;zł</div>
</div>

                <div class="price-instalments-box">
            <div class="price-instalments-row">
                <span class="price-instalments-label">
                    RATY:</span>
                <div class="price-instalments price-instalments-only-shop selenium-price-instalments">
                            274,95 zł <span>x 20 rat 0%</span><i></i>
                            </div>
                    </div>
            <div class="price-instalments-row price-instalment-shop-only-info">
                    Raty tylko w sklepie stacjonarnym<a href="#" class="js-price-installment-shop-info"></a>
                </div>
            </div>
    </div>
</div>

Here is my code:

main_url = 'https://www.euro.com.pl/telefony-komorkowe,_Apple.bhtml'
pages_html = [main_url]
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 10)

driver.get(main_url)

soup = BeautifulSoup(driver.page_source , 'lxml')

pages_numbers = soup.find('div', {'class': 'paging-numbers'})

for page in pages_numbers.find_all('a', {'class': 'paging-number'}):
    print(page['href'])
    pages_html.append('https://www.euro.com.pl' + page['href'])


for url in pages_html:
    driver.get(url)
    soup = BeautifulSoup(driver.page_source, 'lxml')
    product_list = soup.find('div', {"id": "products", 'class': 'list'})

    product_rows = soup.find_all('div', {'class': 'product-row'})

    for row in product_rows:
        product = row.find('h2', class_='product-name')
        all_products.append(product.text.strip())
        try:
            price = row.find('div', {'class': 'price-normal selenium-price-normal'})
        except AttributeError:
            price = row.find('div', {'class': 'price-normal'})
            print(price)

        print(f'Produkt: {product.text.strip()}, Cena: {price.text.strip()}')
driver.close()

Does anyone know what is going on here?

I tried using WebDriverWait object, as other posts suggested but I couln't make it work.

Upvotes: 1

Views: 173

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195428

If you want to get names and prices of products you can use next example:

import pandas as pd
import requests
from bs4 import BeautifulSoup


url = 'https://www.euro.com.pl/telefony-komorkowe,_Apple.bhtml'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

all_data = []
for product in soup.select('.product-for-list'):
    name = product.h2.text.strip()
    price = product.select_one('.price-normal').text.strip()
    all_data.append((name, price))

df = pd.DataFrame(all_data, columns=['Name', 'Price'])
print(df.to_markdown(index=False))

Prints:

Name Price
Apple iPhone 14 128GB (północ) 4 899 zł
Apple iPhone 11 64GB (biały) 2 799 zł
Apple iPhone 14 Pro 128GB (złoty) 7 199 zł
Apple iPhone 11 64GB (czarny) 2 599 zł
Apple iPhone 14 Pro 128GB (gwiezdna czerń) 7 199 zł
Apple iPhone 12‌ 64GB (biały) 3 999 zł
Apple iPhone 13 128GB (księżycowa poświata) 4 749 zł
Apple iPhone 12‌ 64GB (czarny) 3 999 zł
Apple iPhone 13 128GB (północ) 4 199 zł
Apple iPhone 13 128GB (zielony) 4 749 zł
Apple iPhone 13 256GB (księżycowa poświata) 4 999 zł
Apple iPhone 12‌ 64GB (niebieski) 3 999 zł
Apple iPhone 14 Pro 256GB (głęboka purpura) 7 899 zł
Apple iPhone 14 128GB (fioletowy) 4 899 zł
Apple iPhone 14 Pro Max 128GB (złoty) 7 899 zł
Apple iPhone 14 Pro 256GB (srebrny) 7 899 zł
Apple iPhone 14 Pro 128GB (srebrny) 7 199 zł
Apple iPhone 14 Pro 256GB (złoty) 7 899 zł
Apple iPhone 14 Pro Max 1TB (czarny) 11 499 zł
Apple iPhone 14 128GB (księżycowa poświata) 4 899 zł
Apple iPhone 13 128GB (różowy) 4 749 zł
Apple iPhone 12‌ 128GB (biały) 4 549 zł
Apple iPhone 14 Plus 128GB (północ) 5 499 zł
Apple iPhone 12‌ 64GB (fioletowy) 3 299 zł
Apple iPhone 12‌ 128GB (niebieski) 4 549 zł
Apple iPhone 13 mini 128GB (niebieski) 3 499 zł
Apple iPhone 13 mini 128GB (północ) 4 299 zł
Apple iPhone 11 128GB (czarny) 3 099 zł

Upvotes: 1

Related Questions