CODEWITHSUNDEEP
pythonweb-scrapingurllib2
Egor Ignatenkov
Egor Ignatenkov

Reputation: 1322

python urllib2.urlopen returns error 500 while Chrome loads the page

I have a web-page (http://rating.chgk.info/api/tournaments/3506/) I want to open in Python 2 via urllib2. It opens perfectly well in my browser but when I do this:

import urllib2
url = 'http://rating.chgk.info/api/tournaments/3506/'
urllib2.urlopen(url)

i get HTTP Error 500.

I tried tweaking User-Agent and Accept headers but nothing worked. What else can be the problem?

Upvotes: 0

Views: 147

Answers (1)

alecxe
alecxe

Reputation: 474131

You need to first visit the page on the site to get the session cookie set:

In [7]: import requests

In [8]: requests.get("http://rating.chgk.info/api/tournaments/3506")
Out[8]: <Response [500]>

In [9]: with requests.Session() as session:
    ...:     session.get("http://rating.chgk.info/index.php/api")
    ...:     response = session.get("http://rating.chgk.info/api/tournaments/3506")
    ...:     print(response.status_code)
    ...:     
200

Upvotes: 1

Related Questions