Problems with parallel for loop

Question

I am trying to load a full auctionhouse by loading each page async from an api call and putting all the same items together in a list in a dictionary. When I make the parallel for loop, It does not return anything. Help would be appricieted.

Have a great day! -Vexea

    {
        string url = "https://api.hypixel.net/skyblock/auctions";

        //Gets number of pages to make threads on the auction house...
        using (HttpResponseMessage response = await ApiHelper.GetApiClient("application/json").GetAsync(url))
        {
            if (response.IsSuccessStatusCode)
            {
                AuctionHouseModel auctionHouse = await response.Content.ReadAsAsync<AuctionHouseModel>();
                return auctionHouse.Pages;
            }
            else
            {
                return 0;
            }
        }
    }

    private static async Task<AuctionPageModel> LoadHypixelAuctionPage(int page, string apiKey)
    {
        //Loads a solid page...
        string url = "https://api.hypixel.net/skyblock/auctions?key=" + apiKey + "&page=" + page;

        using (HttpResponseMessage response = await ApiHelper.GetApiClient("application/json").GetAsync(url))
        {
            if (response.IsSuccessStatusCode)
            {
                return await response.Content.ReadAsAsync<AuctionPageModel>();
            }
            else
            {
                return null;
            }
        }
    }

    public async static Task<AuctionHouseModel> LoadHypixelAuctionHouse(string apiKey)
    {
        //Loads all pages needed and puts them into a dictionary...
        List<AuctionPageModel> pages = new List<AuctionPageModel>();
        AuctionHouseModel output = new AuctionHouseModel();

        Parallel.For(1, await LoadHypixelAuctionPages(), async page => {
            pages.Add(await LoadHypixelAuctionPage(page, apiKey)); //This returns nothing, count of pages stays 0 and nothing happens...
        });

        foreach (AuctionPageModel page in pages)
            foreach(AuctionProductModel product in page.Products)
                try
                {
                    output.Products[product.Name].Add(product);
                }
                catch
                {
                    output.Products.Add(product.Name, new List<AuctionProductModel>());
                    output.Products[product.Name].Add(product);
                }
        output.Pages = await LoadHypixelAuctionPages();
        return output;
    }

Answer 1

You can't use any Parallel methods with async. Parallel is for CPU-bound code and async is (primarily) for I/O-bound code. The Parallel class doesn't properly understand anything async.

Instead of parallel concurrency, you need asynchronous concurrency (Task.WhenAll):

List<AuctionPageModel> pages = new List<AuctionPageModel>();
AuctionHouseModel output = new AuctionHouseModel();

var tasks = Enumerable
    .Range(1, await LoadHypixelAuctionPages())
    .Select(async page => pages.Add(await LoadHypixelAuctionPage(page, apiKey)))
    .ToList();
await Task.WhenAll(tasks);

or, more simply:

AuctionHouseModel output = new AuctionHouseModel();

var tasks = Enumerable
    .Range(1, await LoadHypixelAuctionPages())
    .Select(async page => await LoadHypixelAuctionPage(page, apiKey))
    .ToList();
var pages = await Task.WhenAll(tasks);

Answer 2

When you're doing parallel programming you need to make sure to use thread-safe types or locking. Perhaps there're more things wrong than this, but the first thing you need to fix is making sure to lock access to the pages list. Secondly, the first paramenter in Parallel.For is inclusive while the second parameter is exclusive. So if LoadHypixelAuctionPages() returns 0 or 1, nothing will run inside the loop, so you probably mean LoadHypixelAuctionPages() + 1 if the first page number is 1 and not 0:

List<AuctionPageModel> pages = new List<AuctionPageModel>();
AuctionHouseModel output = new AuctionHouseModel();

Parallel.For(1, await LoadHypixelAuctionPages() + 1, async page =>
{
    var loadedPage = await LoadHypixelAuctionPage(page, apiKey);
    lock(pages)
    {
        pages.Add(loadedPage);
    }
});

//...

Take a look at this fiddle to see what can happen when not locking.

An alternative to locking is using one of the concurrent collections, such as ConcurrentQueue<T>