CODEWITHSUNDEEP
pythonwebpython-requestsscreen-scraping
Bradley
Bradley

Reputation: 1

requests.get(url) not returning for this specific url

I'm trying to use requests.get(url).text to get the HTML from this website. However, when requests.get(url) is called with this specific url, it never returns no matter how long I wait. This works with other urls, but this one specifically is giving me trouble. Code is below

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.carmax.com/cars/all', allow_redirects=True).text

soup = BeautifulSoup(source, 'lxml')

print(soup.prettify().encode('utf-8'))

Thanks for any help!

Upvotes: 0

Views: 623

Answers (1)

Nisit
Nisit

Reputation: 71

Try:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36', "Upgrade-Insecure-Requests": "1","DNT": "1","Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language": "en-US,en;q=0.5","Accept-Encoding": "gzip, deflate"}
html = requests.get("https://www.carmax.com/cars/all",headers=headers)
soup = BeautifulSoup(html.content, 'html.parser')
print(soup.prettify())

Upvotes: 1

Related Questions