CODEWITHSUNDEEP
pythonhttpurl
rob bruckmeier
rob bruckmeier

Reputation: 1

using python urlopen for a url query

Using urlopen also for url queries seems obvious. What I tried is:

import urllib2
query='http://www.onvista.de/aktien/snapshot.html?ID_OSI=86627'
f = urllib2.urlopen(query)
s = f.read()
f.close()

However, for this specific url query it fails with HTTP error 403 forbidden When entering this query in my browser, it works. Also when using http://www.httpquery.com/ to submit the query, it works.

Do you have suggestions how to use Python right to grab the correct response?

Upvotes: 0

Views: 1685

Answers (2)

Anonymous Coward
Anonymous Coward

Reputation: 6226

It appears that the urllib2 default user agent is banned by the host. You can simply supply your own user agent string:

import urllib2
url = 'http://www.onvista.de/aktien/snapshot.html?ID_OSI=86627'
request = urllib2.Request(url, headers={"User-Agent" : "MyUserAgent"})
contents = urllib2.urlopen(request).read()
print contents

Upvotes: 2

Jon Clements
Jon Clements

Reputation: 142216

Looks like it requires cookies... (which you can do with urllib2), but an easier way if you're doing this, is to use requests

import requests
session = requests.session()
r = session.get('http://www.onvista.de/aktien/snapshot.html?ID_OSI=86627')

This is generally a much easier and less-stressful method of retrieving URLs in Python.

requests will automatically store and re-use cookies for you. Creating a session is slightly overkill here, but is useful for when you need to submit data to login pages etc..., or re-use cookies across a site... etc...

using urllib2 is something like

import urllib2, cookielib

cookies = cookielib.CookieJar()
opener = urllib2.build_opener( urllib2.HTTPCookieProcessor(cookies) )
data = opener.open('url').read()

Upvotes: 3

Related Questions