CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
ahmed mani
ahmed mani

Reputation: 301

beautifulsoup select method returns traceback

well im still learning beautifulsoup module and im replcating this from the book automate the boring stuff with python i tried replcating the get amazon price script but i get a traceback on the .select() method the error 'TypeError: 'NoneType' object is not callable' its getiing devastated with this error as i couldnt find much about it 

import bs4
import requests


header = {'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}

def site(url):

    x = requests.get(url, headers=header)
    x.raise_for_status()
    soup = bs4.BeautifulSoup(x.text, "html.parser")
    p = soup.Select('#buyNewSection > a > h5 > div > div.a-column.a-span8.a-text-right.a-span-last > div > span.a-size-medium.a-color-price.offer-price.a-text-normal')
    abc = p[0].text.strip()
    return abc

price = site('https://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994')
print('price is' + str(price))

it must return a list value containing the price but im stuck with this error

Upvotes: 1

Views: 199

Answers (1)

rrcal
rrcal

Reputation: 3752

If you use soup.select as opposed to soup.Select, your code does work, it just returns an empty list.The reason can see if we inspect the function you are using:

help(soup.Select)

Out[1]:
Help on NoneType object:

class NoneType(object)
 |  Methods defined here:
 |  
 |  __bool__(self, /)
 |      self != 0
 |  
 |  __repr__(self, /)
 |      Return repr(self).
 |  
 |  ----------------------------------------------------------------------
 |  Static methods defined here:
 |  
 |  __new__(*args, **kwargs) from builtins.type
 |      Create and return a new object.  See help(type) for accurate signature.

Compared to:

help(soup.select)

Out[2]:
Help on method select in module bs4.element:

select(selector, namespaces=None, limit=None, **kwargs) method of bs4.BeautifulSoup instance
    Perform a CSS selection operation on the current element.

    This uses the SoupSieve library.

    :param selector: A string containing a CSS selector.

    :param namespaces: A dictionary mapping namespace prefixes
    used in the CSS selector to namespace URIs. By default,
    Beautiful Soup will use the prefixes it encountered while
    parsing the document.

    :param limit: After finding this number of results, stop looking.

    :param kwargs: Any extra arguments you'd like to pass in to
    soupsieve.select().

Having that said, it seems that the page structure is actually different than the one you are trying to get, missing the <a> tag.

<div id="buyNewSection" class="rbbHeader dp-accordion-row">
   <h5>
      <div class="a-row">
         <div class="a-column a-span4 a-text-left a-nowrap">
            <span class="a-text-bold">Buy New</span>
         </div>
         <div class="a-column a-span8 a-text-right a-span-last">
            <div class="inlineBlock-display">
               <span class="a-letter-space"></span>
               <span class="a-size-medium a-color-price offer-price a-text-normal">$16.83</span>
            </div>
         </div>
      </div>
   </h5>
</div>

So this should work:

p = soup.select('#buyNewSection > h5 > div > div.a-column.a-span8.a-text-right.a-span-last > div.inlineBlock-display > span.a-size-medium.a-color-price.offer-price.a-text-normal')
abc = p[0].text.strip()
abc

Out[2]:
'$16.83'

Additionally, you could consider using a more granular approach that let's you debug your code better. For instance:

buySection = soup.find('div', attrs={'id':'buyNewSection'})
buySpan = buySection.find('span', attrs={'class': 'a-size-medium a-color-price offer-price a-text-normal'})

print (buyScan)
Out[1]:
'$16.83'

Upvotes: 1

Related Questions