CODEWITHSUNDEEP
c#asynchronous
VSO
VSO

Reputation: 12636

Is Breaking Up Large Batches Into Async Operations Faster?

Let's say I have an array of ten million items. I want to do some operation on each item in a foreach loop and then return that item. 

foreach(var item in items)
{
  /Lets pretend this is resource-intensive
  item.someIntProp++;
}

Would breaking up the ten million item into, say, 100k item batches and then running each batch in an async operation be any faster?

The actual scenario is needing to map a bunch of objects from mongoDb bson values into .NET objects using automapper. No database calls are made during this process, .NET is just converting bsonString to string, etc.

On one hand it would seem to me that "Yes, it will be faster because multiple batches will be handled simultaneously rather than in order." On the other hand, that seems ridiculous that it wouldn't already be optimizing that.

Upvotes: 1

Views: 176

Answers (2)

weir
weir

Reputation: 4761

Using a Partitoner, it is easy to experiment with various chunk sizes into which your array is divided before the chunks are processed in parallel. An example is given here:

How can I maximize the performance of element-wise operation on an big array in C#

Upvotes: 0

Ryan Ternier
Ryan Ternier

Reputation: 8804

To quickly answer your question: Yes, but don't do it the way you posted.

Here's a good question to read over to give yourself some information on this. Optimal number of threads per core

If you have a requirement to process that many resource intensive operations I would suggest creating a new system to manage them. When you have this many asynchronous processes, you're going to have a lot of context switching, which is going to slow this down.

That being said, if this is a singular application where you just need to run it to do some conversions, don't blindly throw things to an Async Process. Use the Task Parallel Libraries. Doing so will let you manage the tasks, and let you tweak the performance based on a set of inputs (Max/min tasks|threads, current # of items in process, etc.).

This will allow you to figure out the best settings for your app, and from there you can re-use this code for other scenarios where you need batch jobs.

Upvotes: 1

Related Questions