CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Rakesh Majhi
Rakesh Majhi

Reputation: 159

Getting 'NoneType' object has no attribute 'text' error with BeautifulSoup

I am new to the python and dyjango. I am trying to make a scraper using BeautifulSoup. My site has a search form where user searches for products that am using BeautifulSoup to scrape amazon.in to show the results to the user.

Everything works fine but it gives me two types of error sometime it gives me the error:

File "C:\Users\Ahmed\Anaconda3\lib\site-packages\django\db\backends\sqlite3\base.py", line 383, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: NOT NULL constraint failed: myapp_search.search

The above exception was the direct cause of the following exception:

and sometimes it gives the error:

File "C:\Users\Ahmed\codelist\myapp\views.py", line 24, in new_search
    post_title = post.find(class_='a-section').text
AttributeError: 'NoneType' object has no attribute 'text'
[15/Mar/2020 13:34:15] "POST /new_search HTTP/1.1" 500 87165

my views.py

import requests
from bs4 import BeautifulSoup
from django.shortcuts import render
from requests.compat import quote_plus
from . import models

BASE_AMAZON_URL = 'https://www.amazon.in/s?k={}'

def home(request):
    return render(request, 'base.html')

def new_search(request):
    search = request.POST.get('search')
    models.Search.objects.create(search=search)
    final_url = BASE_AMAZON_URL.format(quote_plus(search))
    response = requests.get(final_url)
    data = response.text
    soup = BeautifulSoup(data, features='html.parser')

    post_listings = soup.find_all('', {'class': 's-result-list'})

    final_postings = []
    for post in post_listings:
        post_title = post.find(class_='a-section').text
        post_url = post.find('a').get('href')
        post_price = post.find(class_='a-price').text

        final_postings.append((post_title, post_url, post_price))

    stuff_for_frontend = {
    'search': search,
    'final_postings': final_postings,
    }
    return render(request, 'myapp/new_search.html', stuff_for_frontend)

my new_search.html

{% extends 'base.html' %}

{% block content %}
    <h2 style="text-align: center;">{{ search | title }}</h2>

    {% for post in final_postings %}
    <p>{{ post.0 }}</p>
    <p>{{ post.1 }}</p>
    <p>{{ post.2 }}</p>
    {% endfor %}


{% endblock content %}

Upvotes: 0

Views: 4431

Answers (1)

GordonAitchJay
GordonAitchJay

Reputation: 4860

As per the Beautiful Soup Documentation:

AttributeError: 'NoneType' object has no attribute 'foo' - This usually happens because you called find() and then tried to access the .foo attribute of the result. But in your case, find() didn’t find anything, so it returned None, instead of returning a tag or a string. You need to figure out why your find() call isn’t returning anything.

It could not find an element you searched for at this line, and returned None: post_title = post.find(class_='a-section').text

Upvotes: 3

Related Questions