CODEWITHSUNDEEP
c#arrayscollectionsforeachienumerable
Matt
Matt

Reputation: 7254

What is slowing down simple collection conversions

I have a simple while loop

IEnumerable<Foo> collection;
while (!bc.IsCompleted)
{
   collection = bc.Take();
}

bc is a BlockingCollection<IEnumerable<Foo>>. The bc contains 9 IEnumerable collections and a total of 2.6million Foo objects. The loop takes about 640ms to run on my machine. As soon as I add a foreach loop after Take() within the while loop the time it takes to run explodes to 2400ms.

foreach(Foo foo in collection)
{
}

Iterating over 2.6 million elements within a List or Foo[] or IEnumerable that I setup separately, took about 54ms.

The same happens if instead of the foreach loop I simply add a collection conversion such as 

List<Foo> fooList = collection.ToList();

or

Foo[] fooArray = collection.ToArray();

It suddenly also takes north of 2000ms to execute.

How can this be? I am completely running out of explanations or possible reasons. Anyone who could point me to what I am missing here? The slow down cannot be caused by locking/blocking because I do not alter the way the BlockingCollection is accessed between my comparisons.

Thanks for any input.

Upvotes: 2

Views: 149

Answers (3)

YavgenyP
YavgenyP

Reputation: 2123

As with others LINQ methiods out there (and im guessing you are using LINQ's take), this method works with a deffered execution:

This method is implemented by using deferred execution. The immediate return value is an object that stores all the information that is required to perform the action. The query represented by this method is not executed until the object is enumerated either by calling its GetEnumerator method directly or by using foreach in Visual C# or For Each in Visual Basic

This means that if you dont add the ToList() call of the foreach loop, the sole call to Take doesnt really produce any results, and the actuall results are only brought when you are using the iterator (of foreach/tolist), hence the performance difference.

Your comparison to just iterating over list might not provide accurate results; Its not the iteration over the List(foo) that takes time, its probably the selection of the elements from the blocking collection you are using that slows everything down.
MSDN claims that using regular foreach on the BlockingCollection (which is probably what happens when u r using the LINQ supplied Take, which works on IEunmerable in this case) uses a snashop of the underlying collection, and this can surely slow done the processing on huge collections.

Upvotes: 1

thecoop
thecoop

Reputation: 46128

An IEnumerable can represent a deferred operation. Sometimes (say, with LINQ, or iterator blocks) the contents of the ienumerable is not actually generated until it is iterated over.

So, your IEnumerable<Foo> might contain enough information to generate the Foos, but not actually do so until you iterate over the enumerable, either in a foreach or using ToList. That's why those operations take a long time.

Upvotes: 1

Daniel
Daniel

Reputation: 16464

What kind of IEnumerable are you putting into the queue?

Keep in mind that LINQ queries are using delayed execution; your code might end up evaluating the query on the consumer thread. Try calling ToList() on the producer thread before you put the element into the queue.

Upvotes: 1

Related Questions