CODEWITHSUNDEEP
pythonhttpweb-scrapingpython-requests
Daniel
Daniel

Reputation: 2004

Python requests url encoding not working

Directly from the python requests examples is this snippet:

payload = {'key1': 'value1', 'key2': 'value2'}  
r = requests.get("http://httpbin.org/get", params=payload)  
print r.url  
u'http://httpbin.org/get?key2=value2&key1=value1'

But when I try to pull data from a website (using requests 0.13.0):

payload = {'one' : 'one', 'two' : 'two' }  
r = requests.get("http://[ip_removed]/clubs/pairs_results/personal.php", params=payload)
print r.url  
http://[ip_removed]/clubs/pairs_results/

That's not correctly encoded at all. Ideas?

EDIT: Looks like the site issues a 302 redirect. How do I deal with that to get the same html that my browser will see.

Upvotes: 1

Views: 3344

Answers (1)

mata
mata

Reputation: 69012

The site simply does a HTTP 302-redirect here which requests follows, so you'll end up on a different url than originally requested. That's not error but the expected behaviour.

If you don't want that, you can add the allow_redirects=False keyword argument when sending the request.

And by the way: a google search with inurl:... quickly reveals which site you're talking about here, even if you remove the ip...

Upvotes: 4

Related Questions