CODEWITHSUNDEEP
pythonhtmlbeautifulsouppython-requestspython-requests-html
Junah201
Junah201

Reputation: 71

When using 'requests.get()', the html is less than the actual html

enter image description here

enter image description here

The first image is the html of the site brought in python, and the second image is the actual html viewed by pressing F12 on the site. I don't know why the two results are different. Other sites display html normally, but I wonder why only that site is not displayed normally.

Here is the Code:

import requests
from bs4 import BeautifulSoup


result = requests.get('https://www.overbuff.com/heroes')

soup = BeautifulSoup(result.text, "html.parser")

print(soup)

Upvotes: 0

Views: 480

Answers (1)

HedgeHog
HedgeHog

Reputation: 25048

Your maybe blocked by the page and should try it with some headers like:

headers = {"user-agent": "Mozilla/5.0"}    
r = requests.get(url, headers=headers)

Example

import requests
from bs4 import BeautifulSoup
url = "https://www.overbuff.com/heroes"
headers = {"user-agent": "Mozilla/5.0"}

r = requests.get(url, headers=headers)

soup = BeautifulSoup(r.text, "html.parser")

print(soup)

Upvotes: 3

Related Questions