CODEWITHSUNDEEP
pythonhttpweb-scraping
Alex
Alex

Reputation: 540

Receiving 500 HTTP response when posting to website

I am attempting am attempting to extract some information from a website that requires a post to an ajax script.

I am trying to create an automated script however I am consitently running into an HTTP 500 error. This is in contrast to a different data pull I did from a 

url = 'http://www.ise.com/ExchangeDataService.asmx/Get_ISE_Dividend_Volume_Data/'

paramList = ''
paramList += '"' + 'dtStartDate' + '":07/25/2014"'
paramList += ','
paramList += '"' + 'dtEndDate' + '":07/25/2014"';
paramList = '{' + paramList + '}';

response = requests.post(url, headers={
    'Content-Type': 'application/json; charset=UTF-8',
    'data': paramList,
    'dataType':'json'
    })

I was wondering if anyone had any recommendations as to what is happening. This isn't proprietary data as they allow you to manually download it in excel format.

Upvotes: 0

Views: 65

Answers (3)

bruno desthuilliers
bruno desthuilliers

Reputation: 77932

If that's a copy/paste from you actual code, 'data' is probably not supposed to be part of the request headers. As a side note: you don't "post to an ajax script", you post to an URL. The fact that this URL is called via an asynchronous request from some javascript on some page of the site is totally irrelevant.

Upvotes: 1

abarnert
abarnert

Reputation: 366073

The input you're generating is not valid JSON. It looks like this:

{"dtStartDate":07/25/2014","dtEndDate":07/25/2014"}

If you look carefully, you'll notice a missing " before the first 07.

This is one of many reasons you shouldn't be trying to generate JSON by string concatenation. Either build a dict and use json.dump, or if you must, use a multi-line string as a template for str.format or %.

Also, as bruno desthuilliers points out, you almost certainly want to be sending the JSON as the POST body, not as a data header in an empty POST. Doing it the wrong way does happen to work with some back-ends, but only by accident, and that's certainly not something you should be relying on. And if the server you're talking to isn't one of those back-ends, then you're sending the empty string as your JSON data, which is just as invalid.

So, why does this give you a 500 error? Probably because the backend is some messy PHP code that doesn't have an error handler for invalid JSON, so it just bails with no information on what went wrong, so the server can't do anything better than send you a generic 500 error.

Upvotes: 3

Lando
Lando

Reputation: 1

it sounds like a server error. So what your posting could breaking their api due to its formatting. Or their api could be down.

http://pcsupport.about.com/od/findbyerrormessage/a/500servererror.htm

Upvotes: 0

Related Questions