CODEWITHSUNDEEP
c#.nethttpwebrequestwebclienthttpwebresponse
James Lawruk
James Lawruk

Reputation: 31363

DownloadString and HttpWebResponse is not returning the full JSON content

I am trying to store a JSON response from the following url in a string. Most times I run the code, the last several characters of the JSON response are cut off. When you visit the url in a browser, the full JSON response is shown. Sometimes my code works, but most times it returns partial content.

If I reduce the length of the querysting in the url I am requesting, I get more content back and closer to the full response, which is very strange. If I remove the querystring entirely, usually the full JSON response is returned. The problem is I DO want to keep the querystring as is.

I tried all sort of alternatives including changing the encoding, using HttpWebRequest/HttpWebResponse, copying the response into a MemoryStream, reading the response using a byte buffer, changing the protocol version, etc.

Any ideas on how to get the full response and can explain what is happening? Thanks!

System.Net.WebClient wc = new System.Net.WebClient();            
string data = wc.DownloadString("http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback");
Console.Write(data);// String should end with ",-119.2]}]}]});"
Console.Read();

Upvotes: 2

Views: 2302

Answers (3)

ToBeFrank
ToBeFrank

Reputation: 350

Leaving an answer because I don't have enough reputation to leave a comment on your answer...

I think this is a bug in mono on the client side, not a server bug. I've been tracking this down in some code I'm using. It works perfectly on Windows using .Net. It only fails when running the code with mono. When I use your GZipWebClient, it runs perfectly on mono as well.

Upvotes: 0

Justin Grant
Justin Grant

Reputation: 46753

It's a server problem, but I luckily I believe there's a workaround. The server is not returning correct uncompressed results. If you change your client code to accept compressed responses, the problem seems to go away.

What's not clear is why uncompressed responses are broken. It's possible that the server has trouble with responses over 64K (this json is 64.8K). It's possible that the server is has a bug in its length calculations that only shows up in some circumstances or for some maps. I'm not sure. So there's no guarantee that the compression workaround will work in all cases, and I'd recommend testing on your end. And following up the ArcGIS folks so they know about this bug.

Here's the decompression code (adapted from https://stackoverflow.com/a/4567408/126352) that I got to return the correct JSON. 

public class GZipWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
        request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        return request;
    }
}

GZipWebClient wc = new GZipWebClient();
string data = wc.DownloadString("http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback");

Here's how I discovered this workaround:

  • first I reproduced the problem in code
  • then I changed the request to HTTP 1.0 (which doesn't have chunked encoding or compression) and noticed that the Content-Length header returned by the server was 66338. But the actual content is longer than that. So the server is telling the client that the content is shorter than it actually is, which explains why your WebClient response was truncated.
  • then I tried to handle chunked encoding manually with the client, but that didn't work-- I got the same truncation.
  • at that point I wondered if it was an absolute length issue, e.g. the server handled shorter responses OK but underestimated the length of longer ones.
  • so I used Fiddler to play around with the request headers, and I found that removing the compression header (Accept-Encoding:gzip,deflate,sdch) would reproduce the problem.
  • this explained why this request works in a browser (which uses compression) but not in your code.
  • so I tried a compression-aware C# client and everything worked!

Unfortunately, this doesn't tell you the root cause, so the compression workaround may or may not work in all cases. It may break on longer responses. It may break on different maps. Test it to make sure!

It also doesn't explain why @ssut wasn't able to repro. It's possible that only some servers in ArgGIS's cluster are vulnerable? I dunno.

BTW, if you want to report the bug to ArcGIS, here's an HTTP request they can use to repro the problem: 

GET http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback HTTP/1.0
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: static.arcgis.com

Upvotes: 7

ssut
ssut

Reputation: 441

I am tested using your code and this code is working correctly.

p.s. you don't need &callback parameter when you use for just parse json. (callback parameter is need for jsonp callback on javascript)

Upvotes: 0

Related Questions