Async Loop There is no longer an HttpContext available

Question

I have a requirement, is to process X number of files, usually we can receive around 100 files each day, is a zip file so I have to open it, create a stream then send it to a WebApi service which is a workflow, this workflow calls two more WebApi Steps.

I implemented a console application that loops through the files then calls a wrapper which makes a REST call using HttpWebRequest.GetResponse().

I stressed tested the solution and created 11K files, in a synchronous version it takes to process all the files around 17 minutes, but I would like to create an async version of it and be able to use await HttpWebRequest.GetResponseAsync().

Here is the Async version:

private async Task<KeyValuePair<HttpStatusCode, string>> REST_CallAsync(
        string httpMethod,
        string url,
        string contentType,
        object bodyMessage = null,
        Dictionary<string, object> headerParameters = null,
        object[] queryStringParamaters = null,
        string requestData = "")
    {
        try
        {
            HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("some url");
            req.Method = "POST";
            req.ContentType = contentType;

                //Adding zip stream to body
                var reqBodyBytes = ReadFully((Stream)bodyMessage);
                req.ContentLength = reqBodyBytes.Length;
                Stream reqStream = req.GetRequestStream();
                reqStream.Write(reqBodyBytes, 0, reqBodyBytes.Length);
                reqStream.Close();                
            //Async call
            var resp = await req.GetResponseAsync();
            var httpResponse = (HttpWebResponse)resp as HttpWebResponse;
            var responseData = new StreamReader(resp.GetResponseStream()).ReadToEnd();                
            return new KeyValuePair<HttpStatusCode,string>(httpResponse.StatusCode, responseData);
        }
        catch (WebException webEx)
        {
        //something
        }
        catch (Exception ex)
        {
        //something
        }

In my console Application I have a loop to open and call the async (CallServiceAsync under the covers calls the method above)

foreach (var zipFile in Directory.EnumerateFiles(directory))
                                {
                                        using (var zipStream = System.IO.File.OpenRead(zipFile))
                                        {
                                            await _restFulService.CallServiceAsync<WorkflowResponse>(
                                            zipStream,
                                            headerParameters,
                                            null,
                                            true);
                                        }
                                        processId++;
                                    }
                                }

What end up happening was that only 2K of 11K got processed and didn't throw any exception so I was clueless so I changed the version I am calling the async to:

foreach (var zipFile in Directory.EnumerateFiles(directory))
                                {      
                                        using (var zipStream = System.IO.File.OpenRead(zipFile))
                                        {
                                            tasks.Add(_restFulService.CallServiceAsync<WorkflowResponse>(
                                            zipStream,
                                            headerParameters,
                                            null,
                                            true));
                                        }
                                    }
                                }

And have another loop to await for the tasks:

foreach (var task in await System.Threading.Tasks.Task.WhenAll(tasks))
                {
                    if (task.Value != null)
                    {
                        Console.WriteLine("Ending Process");
                    }
                }

And now I am facing a different error, when I process three files, the third one receives:

The client is disconnected because the underlying request has been completed. There is no longer an HttpContext available.

My question is, what i am doing wrong here? I use SimpleInjector as IoC would it be this the problem?

Also when you do WhenAll is waiting for each thread to run? Is not making it synchronous so it waits for a thread to finish in order to execute the next one? I am new to this async world so any help would be really much appreciated.

Answer 1

Well for those that added -1 to my question and instead of providing some type of solution just suggested something meaningless, here it is the answer and the reason why specifying as much detail as possible is useful.

First problem, since I'm using IIS Express if I'm not running my solution (F5) then the web applications are not available, that happened to me sometimes not always.

The second problem and the one giving me a huge headache is that not all the files got processed, I should've known the reason of this issue before, is the usage of async - await in a console application. I forced my console app to work with async by doing:

static void Main(string[] args)
    {
        System.Threading.Tasks.Task.Run(() => MainAsync(args)).Wait();
    }

    static async void MainAsync(string[] args)
    {
      //rest of code

Then if you note in my foreach I had await keyword and what was happening is that by concept await sends back the control flow to the caller, in this case the OS is the one calling the Console App (that is why doesn't make too much sense to use async - await in a console app, I did it because I mistakenly used await by calling an async method). So the result was that my process only processed some X number of files, so what I end up doing is the following:

Add a list of tasks, the same way I did above:

tasks.Add(_restFulService.CallServiceAsync<WorkflowResponse>(....

And the way to run the threads is (in my console app):

ExecuteAsync(tasks);

Finally my method:

static void ExecuteAsync(List<System.Threading.Tasks.Task<KeyValuePair<HttpStatusCode, WorkflowResponse>>> tasks)
    {
         System.Threading.Tasks.Task.WhenAll(tasks).Wait();
    }

UPDATE: Based on Scott's feedback, I changed the way I execute my threads.

And now I'm able to process all my files, I tested it and to process 1000 files in my synchronous process took around 160+ seconds to run all the process (I have a workflow of three steps in order to process the file) and when I put my async process in place it took 80+ seconds so almost half of the time. In my production server with IIS I believe the execution time will be less.

Hope this helps to anyone facing this type of issue.