Restart concurrent tasks as soon as they fail for x number of times

Question

I have a console app that is making HTTP queries and adding/updating products in my database according to response. Some fail and need to be retried a few times.

The way I came up with was to use a dictionary to store the product ID and a Task. Then I can check all the task results and re-run.

This is working but it strikes me as inefficient. Tasks are not being re-created until all tasks have finished. It would be more efficient if they were immediately restarted but I can't figure out how to do this. Also every retry involves a query to the database as only the ID is stored.

I made small app that shows how I am currently retrying failed requests.

Can someone suggest a more efficient method for retrying?

class Program
{
    private static void Main(string[] args)
    {
        HttpQuery m = new HttpQuery();
        var task = Task.Run(() => m.Start());
        Task.WaitAll(task);
        Console.WriteLine("Finished");
        Console.ReadLine();
    }

}
class HttpQuery
{
    public async Task Start()
    {
        // dictionary where key represent reference to something that needs to be processed and bool whether it has completed or not
        ConcurrentDictionary<int, Task<bool>> monitor = new ConcurrentDictionary<int, Task<bool>>();

        // start async tasks. 
        Console.WriteLine("starting first try");
        for (int i = 0; i < 1000; i++)
        {
            Console.Write(i+",");
            monitor[i] = this.Query(i);
        }
        // wait for completion
        await Task.WhenAll(monitor.Values.ToArray());
        Console.WriteLine();
        // start retries
        // number of retries per query
        int retries = 10;
        int count = 0;
        // check if max retries exceeded or all completed
        while (count < retries && monitor.Any(x => x.Value.Result == false))
        {
            // make list of numbers that failed
            List<int> retryList = monitor.Where(x => x.Value.Result == false).Select(x => x.Key).ToList();
            Console.WriteLine("starting try number: " + (count+1) + ", Processing: " + retryList.Count);
            // create list of tasks to wait for
            List<Task<bool>> toWait = new List<Task<bool>>();
            foreach (var i in retryList)
            {
                Console.Write(i + ",");
                monitor[i] = this.Query(i);
                toWait.Add(monitor[i]);
            }

            // wait for completion
            await Task.WhenAll(toWait.ToArray());
            Console.WriteLine();
            count++;
        }

        Console.WriteLine("ended");
        Console.ReadLine();

    }
    public async Task<bool> Query(int i)
    {
        // simulate a http request that may or may not fail
        Random r = new Random();
        int delay = i * r.Next(1, 10);
        await Task.Delay(delay);
        if (r.Next(0,2) == 1)
        {
            return true;
        }
        else
        {
            return false;
        }


    }
}

Answer 1

You can create another method and wrap all these ugly retry logic. All of that ugly code goes away :)

public async Task Start()
{
    const int MaxNumberOfTries = 10;
    List<Task<bool>> tasks = new List<Task<bool>>();
    for (int i = 0; i < 1000; i++)
    {
        tasks.Add(this.QueryWithRetry(i, MaxNumberOfTries));
    }
    await Task.WhenAll(tasks);
}

public async Task<bool> QueryWithRetry(int i, int numOfTries)
{
    int tries = 0;
    bool result;
    do
    {
        result = await Query(i);
        tries++;
    } while (!result && tries < numOfTries);

    return result;
}