HttpWebRequest.BeginGetResponse blocks; What is the correct way to get HttpWebResponse asynchronously?

Question

I'm trying to get an HttpWebResponse object asynchronously, so that I can allow the request to be cancelled (using the BackgroundWorker.CancellationPending flag). However, the request.BeginGetResponse line below blocks until a response is returned (and this can take longer than a minute).

Public Function myGetResponse(ByVal request As HttpWebRequest, ByRef caller As System.ComponentModel.BackgroundWorker) As HttpWebResponse
    'set a flag so that we can wait until our async getresponse finishes
    Dim waitFlag = New ManualResetEvent(False) 

    Dim recieveException As Exception = Nothing

    Dim response As HttpWebResponse = Nothing
    'I use beginGetResponse here and then wait for the result, rather than just using getResponse
    'so that this can be aborted cleanly.
    request.BeginGetResponse(Sub(result As IAsyncResult)
                                 Try 'catch all exceptions here, so they can be raised on the outer thread
                                     response = request.EndGetResponse(result)
                                 Catch e As Exception
                                     recieveException = e
                                 End Try

                                 waitFlag.Set()
                             End Sub, Nothing)


    While Not waitFlag.WaitOne(100) AndAlso Not caller.CancellationPending
        'check for cancelation every 100ms
    End While


    'if our async read from the server returned an exception, raise it here. 
    If recieveException IsNot Nothing Then
        Throw recieveException
    End If

    Return response
End Function

The MSDN documentation for BeginGetResponseincludes the paragraph,

The BeginGetResponse method requires some synchronous setup tasks to complete (DNS resolution, proxy detection, and TCP socket connection, for example) before this method becomes asynchronous. As a result, this method should never be called on a user interface (UI) thread because it might take some time, typically several seconds. In some environments where the webproxy scripts are not configured properly, this can take 60 seconds or more. The default value for the downloadTime attribute on the config file element is one minute which accounts for most of the potential time delay.

Am I doing something wrong, or is it likely these initial synchronous tasks that are causing the delay? If the latter, how do I make this call actually asynchronous?

This answer seems to suggest a possible .NET bug, but I know the URL I'm using is valid (I'm running a dev server locally)

I'm using VS 2010 and .NET 4.0

Answer 1

Something I've run into before that might be it, although this is really just a stab in the dark.

You don't show how you're configuring the HttpWebRequest object. If this is a POST request, then you have to use BeginGetRequestStream to send the data. As the documentation for HttpWebRequest.BeginGetResponse says:

Your application cannot mix synchronous and asynchronous methods for a particular request. If you call the BeginGetRequestStream method, you must use the BeginGetResponse method to retrieve the response.

Answer 2

The MSDN docs are correct, there's indeed a synchronous part which may block the UI if HttpWebRequest.BeginGetResponse or HttpWebRequest.GetResponse is called on the UI thread.

Thus, you'd need to call it on a separate thread. I'd suggest using Task Parallel Library for that. Task.Factory.FromAsync could help wrapping BeginGetResponse/EndGetResponse, then you'd use Task.Factory.StartNew, Task.Unwrap and Task.ContinueWith to start the request on pool thread and handle its completion. This can be further enhanced to support cancellation.

I'm not good enough with VB.NET to create a sample of this, but here it is in C# (.NET 4.0, VS2010), untested. Shouldn't be a problem to convert this to VB.NET, the API is the same.

public static class WebResponseExt
{
    // convert BeginGetResponse/EndGetResponse into a Task
    static public Task<WebResponse> GetResponseTapAsync(
        this WebRequest request)
    {
        return Task.Factory.FromAsync(
             (asyncCallback, state) =>
                 request.BeginGetResponse(asyncCallback, state),
             (asyncResult) =>
                 request.EndGetResponse(asyncResult), null);
    }

    // call GetResponseTapAsync on a pool thread 
    // and unwrap the nested task
    static public Task<WebResponse> OffloadGetResponseTapAsync(
        this WebRequest request, CancellationToken token)
    {
        return Task.Factory.StartNew(
            () => request.GetResponseTapAsync(),
            token,
            TaskCreationOptions.None,
            TaskScheduler.Default).Unwrap();
    }
}

class Program
{
    void DoRequest(string url, CancellationToken token)
    {
        var request = HttpWebRequest.Create(url);

        var cancellation = token.Register(() => request.Abort());

        var task1 = request.OffloadGetResponseTapAsync(token);

        var task2 = task1.ContinueWith(
            (responseTask) =>
            {
                cancellation.Dispose();
                try
                {
                    WebResponse result = responseTask.Result;
                    // process result
                }
                catch (Exception ex)
                {
                    Debug.Print(ex.ToString());
                    // process 

                    // rethrow if needed
                    // the exception will be available as task2.Exception
                    throw; 
                }
            },
            token,
            TaskContinuationOptions.None,
            TaskScheduler.FromCurrentSynchronizationContext());
    }

    // ...
}

If you decide to go on with Task-based API, consider using task composition patterns like Stephen Toub's Then.