Can't get ResponseStream from WebException

Question

I have a desktop client, that communicates with serverside via Http. When server has some issues with data processing it returns description of an error in JSON in Http response body with proper Http-code (mainly it is HTTP-400).

When i read HTTP-200 response everithing's fine and this code works:

 using (var response = await httpRequest.GetResponseAsync(token))
                {
                    using (var reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("utf-8")))
                    {
                        return await reader.ReadToEndAsync();
                    }
                }

But when an error occures and WebException is thrown and caught there is this code:

 catch (WebException ex)
            {
                 if (ex.Status == WebExceptionStatus.ProtocolError)
                {                   
                    using (var response = (HttpWebResponse) ex.Response)
                    {
                        using (var stream = response.GetResponseStream())
                        {
                            using (var reader = new StreamReader(stream, Encoding.GetEncoding("utf-8")))
                            {
                                var json = reader.ReadToEnd();
                            }
                        }
                    }
                }
            }

I have already done something to it to maybe make it work, but the next happens: response.ContentLength is valid (184) but stream.Length is 0 and after that i can't read json (it's "") I don't even know where to look, because everything looks like it should work.

What might be the problem?

Answer 1

After 9 years of the original question, I was here facing exactly the same problem. I hope my investigation can help others.

As mentioned by @maiksaray we need to be careful with HttpWebRequest.DefaultMaximumErrorResponseLength.

Microsoft documentation does nos says that it's using kilobytes (and not bytes) values. So the property value is multiplied by 1024 here: https://github.com/microsoft/referencesource/blob/master/System/net/System/Net/HttpWebRequest.cs - leading to a negative result when Int32.MaxValue is applied, because it's a signed Int32.

To avoid the problem, the maximum safe value for this property is Int32.MaxValue / 1024. In practical terms, don't use nothing greater than 1024 * 1024 (1 GB when MS code multiplies again by 1024).

I opened a feedback on MS Documentation and I believe the docs will be fixed.

Answer 2

After a month of almost everyday thinking I've found workaround.

The thing was that WebException.Response.GetResponseStream() returns not exactly the same stream that was obtained during request (can't find link to msdn right now) and by the time we get to catch the exception and read this stream the actual response stream is lost (or something like that, don't really know and was unable to find any info on the net, except looking into CLRCore which is now opensource).

To save the actual response until catching WebException you must set KeepAlive Property on your HttpRequest and voila, you get your response while catching exception. So the working code looks like that:

            try
            {
                var httpRequest = WebRequest.CreateHttp(Protocol + ServerUrl + ":" + ServerPort + ServerAppName + url);

                if (HttpWebRequest.DefaultMaximumErrorResponseLength < int.MaxValue)
                    HttpWebRequest.DefaultMaximumErrorResponseLength = int.MaxValue;

                httpRequest.ContentType = "application/json";
                httpRequest.Method = method;
                var encoding = Encoding.GetEncoding("utf-8");

                if (httpRequest.ServicePoint != null)
                {
                    httpRequest.ServicePoint.ConnectionLeaseTimeout = 5000;
                    httpRequest.ServicePoint.MaxIdleTime = 5000;
                }
                //----HERE--
                httpRequest.KeepAlive = true;
                //----------
                using (var response = await httpRequest.GetResponseAsync(token))
                {
                    using (var reader = new StreamReader(response.GetResponseStream(), encoding))
                    {                       
                        return await reader.ReadToEndAsync();
                    }
                }
            }
            catch (WebException ex)
            {
                if (ex.Status == WebExceptionStatus.ProtocolError)
                {
                    using (var response = (HttpWebResponse)ex.Response)
                    {
                        using (var stream = response.GetResponseStream())
                        {
                            using (var reader = new StreamReader(stream, Encoding.GetEncoding("utf-8")))
                            {
                                return reader.ReadToEnd();
                                //or handle it like you want
                            }
                        }
                    }
                }
            }

I don't know if it is good to keep all connection alive like that, but since it helped me to read actual responses from server, i think it might help someone, who faced the same problem.

EDIT: Also it is important not to mess with HttpWebRequest.DefaultMaximumErrorResponseLength.

Answer 3

I remember facing similar issue before and there was something related to setting the stream's position. Here is one of my solutions for reading webResponse that worked for me earlier. Please try if similar approach works for you:-

private ResourceResponse readWebResponse(HttpWebRequest webreq)
    {
        HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;
        HttpWebResponse webresp = null;// = webreq.GetResponse() as HttpWebResponse;
        var memStream = new MemoryStream();
        Stream webStream;
            try
            {
                webresp = (HttpWebResponse)webreq.GetResponse();
                webStream = webresp.GetResponseStream();
                byte[] readBuffer = new byte[4096];
                int bytesRead;
                while ((bytesRead = webStream.Read(readBuffer, 0, readBuffer.Length)) > 0)
                    memStream.Write(readBuffer, 0, bytesRead);
            }
            catch (WebException e)
            {
                var r = e.Response as HttpWebResponse;
                webStream = r.GetResponseStream();
                memStream = Read(webStream);
                var wrongLength = memStream.Length;
            }


            memStream.Position = 0;
            StreamReader sr = new StreamReader(memStream);
            string webStreamContent = sr.ReadToEnd();

            byte[] responseBuffer = Encoding.UTF8.GetBytes(webStreamContent);
//......
//.......

Hope this helps!