CODEWITHSUNDEEP
c#.nethttpwebrequestmtgox
symbiont
symbiont

Reputation: 1562

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:

https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>

documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades

here's the function:

string GetTradesOnline(Int64 tid)
{
    Thread.Sleep(30000);

    // communicate
    string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    StreamReader reader = new StreamReader(response.GetResponseStream());

    string json = reader.ReadToEnd();
    reader.Close();
    reader.Dispose();
    response.Close();

    return json;
}

i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

here are the facts:

any ideas of what could be wrong?

Upvotes: 31

Views: 100681

Answers (5)

symbiont
symbiont

Reputation: 1562

This is not a solution, but just an alternative: These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.

Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.

This is how i would do it now

string GetTradesOnline(Int64 tid)
{
    using (var wc = new WebClient())
    {
        return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
    }
}

2 more POST examples

// POST
string SubmitData(string data)
{
    string response;
    using (var wc = new WebClient())
    {
        wc.Headers["Content-type"] = "text/plain";
        response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
    }

    return response;
}

// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
    // url encoded query
    NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
    query.Add("project", project);
    query.Add("subject", subject);

    // url encoded data
    NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
    data.Add("sender", sender);
    data.Add("message", message);

    string response;
    using (var wc = new WebClient())
    {
        wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
        response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
                                  , WebRequestMethods.Http.Post
                                  , data.ToString()
                                  );
    }

    return response;
}

Error handling

try
{
    Console.WriteLine(GetTradesOnline(0));

    string data = File.ReadAllText(@"C:\mydata.txt");
    Console.WriteLine(SubmitData(data));

    Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
    string msg;
    if (ex.Response != null)
    {
        // read response HTTP body
        using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
    }
    else
    {
        msg = ex.Message;
    }

    Log(msg);
}

Upvotes: 2

bech
bech

Reputation: 637

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

Upvotes: 1

Habeeb
Habeeb

Reputation: 8017

I had the very same issue. For me the fix was as simple as wrapping the HttpWebResponse code in using block.

using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
    // Do your processings here....
}

Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

Upvotes: 46

Karl
Karl

Reputation: 415

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.

Here is my final method for calling the REST API.

        private static string RunWebRequest(string url, string json)
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

        // Header
        request.ContentType = "application/json";
        request.Method = "POST";
        request.AllowAutoRedirect = false;
        request.KeepAlive = false;
        request.Timeout = 30000;
        request.ReadWriteTimeout = 30000;
        request.UserAgent = "test.net";
        request.Accept = "application/json";
        request.ProtocolVersion = HttpVersion.Version11;
        request.Headers.Add("Accept-Language","de_DE");
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
        ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
        byte[] bytes = Encoding.UTF8.GetBytes(json);
        request.ContentLength = bytes.Length;
        using (var writer = request.GetRequestStream())
        {
            writer.Write(bytes, 0, bytes.Length);
            writer.Flush();
            writer.Close();
        }

        var httpResponse = (HttpWebResponse)request.GetResponse();
        using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
        {
            var jsonReturn = streamReader.ReadToEnd();
            return jsonReturn;
        }
    }

Upvotes: 10

kevin c
kevin c

Reputation: 805

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:

request.Timeout = Timeout.Infinite;
request.KeepAlive = true;

Try something like this...

Upvotes: 23

Related Questions