The underlying connection was closed exception while WebClient DownloadString

Question

Just a piece of code

WebClient wc = new WebClient();
String str = wc.DownloadString(new Uri("http://content.warframe.com/dynamic/rss.php"));

And I got exception:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The underlying connection was closed: The connection was closed unexpectedly.

I've head that this is a bug in .NET (I am using 3.5), but I tried other methods to obtain the content of this link (its rss, xml). No lucky shot yet

var webrequest = (WebRequest)HttpWebRequest.Create(@"http://content.warframe.com/dynamic/rss.php");
var resp = webrequest.GetResponse();
//HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse(); // Wont work also

This code above won't work either, both casts the same exception

Fiddler logs:

SESSION STATE: Aborted.
Response Entity Size: 512 bytes.

== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler] 0x100
X-ABORTED-WHEN: Done
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 2747
X-EGRESSPORT: 2748
X-FAILSESSION-WHEN: ReadingResponse
X-HOSTIP: 205.185.216.10
X-PROCESSINFO: willitwork.vshost:3300

== TIMING INFO ============
ClientConnected: 10:29:11.706
ClientBeginRequest: 10:29:11.713
GotRequestHeaders: 10:29:11.713
ClientDoneRequest: 10:29:11.713
Determine Gateway: 0ms
DNS Lookup: 164ms
TCP/IP Connect: 74ms
HTTPS Handshake: 0ms
ServerConnected: 10:29:11.953
FiddlerBeginRequest: 10:29:11.953
ServerGotRequest: 10:29:11.953
ServerBeginResponse: 10:29:12.372
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 10:29:12.372
ClientBeginResponse: 10:29:12.385
ClientDoneResponse: 10:29:12.385

Overall Elapsed: 0:00:00.672

The response was buffered before delivery to the client.

== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.

Also - 504, this does not makes sense since I can get data from link via chrome / firefox / ie...

I just did it to work in other language, but I am forced to do it with C# (I' ve made 2 much code to rewrite it)

I've added some settings like fiddler said

myHttpWebRequest1.ProtocolVersion = HttpVersion.Version11;
myHttpWebRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36";
myHttpWebRequest1.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

At least now I get 504 error instead of "unknown", but I can still view the content via webbrowser, so the 504 error is fake

Edit: There is no response error when I added

myHttpWebRequest1.Headers["Accept-Encoding"] = "gzip";

but now the output is messed and unreadable

Answer 1

Check this answer:

..The underlying connection was closed: An unexpected error occurred on a receive

So this may work for you:

var webRequest = (HttpWebRequest)HttpWebRequest.Create(@"http://content.warframe.com/dynamic/rss.php");
webRequest.KeepAlive = false;
var resp = webRequest.GetResponse();        

EDIT:

You are right, check rather this: http://msdn.microsoft.com/cs-cz/library/system.net.httpwebrequest.keepalive%28v=vs.110%29.aspx

Here is working code that will print out the recieved response content:

    static void Main(string[] args)
    {
          // Create a new HttpWebRequest object.Make sure that 
          // a default proxy is set if you are behind a firewall.
          HttpWebRequest myHttpWebRequest1 =
            (HttpWebRequest)WebRequest.Create(@"http://content.warframe.com/dynamic/rss.php");

          myHttpWebRequest1.KeepAlive=false;
          // Assign the response object of HttpWebRequest to a HttpWebResponse variable.
          HttpWebResponse myHttpWebResponse1 = 
            (HttpWebResponse)myHttpWebRequest1.GetResponse();

          Console.WriteLine("\nThe HTTP request Headers for the first request are: \n{0}", myHttpWebRequest1.Headers);

          Stream streamResponse = myHttpWebResponse1.GetResponseStream();
          StreamReader streamRead = new StreamReader(streamResponse);
          Char[] readBuff = new Char[256];
          int count = streamRead.Read(readBuff, 0, 256);
          Console.WriteLine("The contents of the Html page are.......\n");
          while (count > 0)
          {
              String outputData = new String(readBuff, 0, count);
              Console.Write(outputData);
              count = streamRead.Read(readBuff, 0, 256);
          }
          Console.WriteLine();
          // Close the Stream object.
          streamResponse.Close();
          streamRead.Close();
          Console.ReadKey();
    }

Answer 2

I have same error.

You can add User Agent to your httpRequest

request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";

Answer 3

Ok, i got this all fixes & working!

static void Main(string[] args)
{
    Uri url = new Uri(@"http://content.warframe.com/dynamic/rss.php");

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

    // MAGIC LINE GOES HERE \/
    request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

    // Assign the response object of HttpWebRequest to a HttpWebResponse variable.
    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
    {
        using (Stream streamResponse = response.GetResponseStream())
        {
            using (StreamReader streamRead = new StreamReader(streamResponse))
            {
                Char[] readBuff = new Char[2000];
                int count = streamRead.Read(readBuff, 0, 2000);
                while (count > 0)
                {
                    String outputData = new String(readBuff, 0, count);
                    Console.Write(outputData);
                    count = streamRead.Read(readBuff, 0, 2000);
                }
            }
        }
    }

    Console.ReadKey();
}

Besides of not-using WebClient.DownloadString method i had to add decompresion line

request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

Thanks for tips (especially fiddler one, Decode button saved my time to find what's wrong)