CODEWITHSUNDEEP
pythonbeautifulsoup
Abdullah Munir
Abdullah Munir

Reputation: 23

BeautifulSoup using Python keep returning null even though the element exists

I am running the following code to parse an amazon page using beautiful soup in Python but when I run the print line, I keep getting None. I am wondering whether I am doing something wrong or if theres an explanation/solution to this. Any help will be appreciated.

    import requests
    from bs4 import BeautifulSoup

    URL = 'https://www.amazon.ca/Magnetic-Erase-Whiteboard-Bulletin- 
    Board/dp/B07GNVZKY2/ref=sr_1_3_sspa?keywords=whiteboard&qid=1578902710&s=office&sr=1-3-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEzOE5ZSkFGSDdCOFVDJmVuY3J5cHRlZElkPUEwMDM2ODA4M0dWMEtMWkI1U1hJJmVuY3J5cHRlZEFkSWQ9QTA0MDIwMjQxMEUwMzlMQ0pTQVlBJndpZGdldE5hbWU9c3BfYXRmJmFjdGlvbj1jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ=='

    headers = {"User-Agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'}

    page = requests.get(URL, headers=headers)

    soup = BeautifulSoup(page.content, 'html.parser')

    title = soup.find(id="productTitle")

    print(title)

Upvotes: 1

Views: 289

Answers (2)

Suyash
Suyash

Reputation: 420

Your code is absolutely correct. There seems to be some issue with the the parser that you have used (html.parser)

I used html5lib in place of html.parser and the code now works:

import requests
from bs4 import BeautifulSoup

URL = 'https://www.amazon.ca/Magnetic-Erase-Whiteboard-BulletinBoard/dp/B07GNVZKY2/ref=sr_1_3_sspa?keywords=whiteboard&qid=1578902710&s=office&sr=1-3-spons&psc=1&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEzOE5ZSkFGSDdCOFVDJmVuY3J5cHRlZElkPUEwMDM2ODA4M0dWMEtMWkI1U1hJJmVuY3J5cHRlZEFkSWQ9QTA0MDIwMjQxMEUwMzlMQ0pTQVlBJndpZGdldE5hbWU9c3BfYXRmJmFjdGlvbj1jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ=='

headers = {"User-Agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html5lib')

title = soup.find(id='productTitle')

print(title)

More Info not directly related to the answer:

For the other answer given to this question, I wasn't asked for a captcha when visiting the page.

However Amazon does change the response content if it detects that a bot is visiting the website: Remove the headers from requests.get() method, and try page.text

The default headers added by requests library lead to the identification of the request as being form a bot.

Upvotes: 1

Jack
Jack

Reputation: 5614

When requesting that page outside of a normal browser environment it asked for a captcha, I'd assume that's why the element doesn't exist.

Amazon probably has specific measures to counter "robots" accessing their pages, I suggest to look at their APIs to see if there's anything helpful instead of scraping the webpages directly.

Upvotes: 0

Related Questions