CODEWITHSUNDEEP
pythonpython-requestsurllib
andy
andy

Reputation: 62

Difference between requests and urllib

UPD: no, the linked question is not the same and doesn't answer mine (unless i'm illiterate).
it's more of a surface question on why there are so many request-making libraries in python and which one to use. i'm fully aware of the options and my question is a little more specific

What is the difference in requests between the requests module and the urllib module?

For testing purposes, i'm sending a GET request to
https://www.securityweek.com/
with just a user-agent header.
urllib gets the correct page as a response while requests gets a response with "Enable JavaScript and cookies to continue"

Here are the examples:
using requests:

import requests


url = 'https://www.securityweek.com/'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

response = requests.get(url, headers=headers)
with open('test1.html', 'w', encoding='utf-8') as f:
    f.write(response.text)

using urllib:

from urllib.request import Request, urlopen


url = 'https://www.securityweek.com/'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

response = urlopen(Request(url, headers=headers))
with open('test2.html', 'w', encoding='utf-8') as f:
    f.write(response.read().decode('utf-8'))

I've tried checking with tools like httpbin but as far as I can see they should be the same.

Upvotes: -1

Views: 160

Answers (1)

Mario Mateaș
Mario Mateaș

Reputation: 1216

requests typically uses urllib underneath, but it tends to provide an easier to use API when sending requests to an endpoint.

Regarding your question, neither requests nor urllib execute Javascript (Selenium could be picked for this because it uses a literal web driver), but it might differ on how those two actually bypass bot detection systems in various websites.

requests automatically sends a default set of headers, including User-Agent, Accept-Encoding and Detection, which could trigger those bot systems and block the further interaction with the web server. Keep-Alive & Connection Pooling doesn't help either.

More specific differences between the two can be found on this topic on the website.

Upvotes: 0

Related Questions