CODEWITHSUNDEEP
c#.netlinq
Bob
Bob

Reputation: 387

LINQ: take a sequence of elements from a collection

I have a collection of objects and need to take batches of 100 objects and do some work with them until there are no objects left to process.

Instead of looping through each item and grabbing 100 elements then the next hundred etc is there a nicer way of doing it with linq?

Many thanks

Upvotes: 4

Views: 4664

Answers (5)

Tom Ferguson
Tom Ferguson

Reputation: 917

var batchSize = 100;
for (var i = 0; i < Math.Ceiling(yourCollection.Count() / (decimal)batchSize); i++)
{
    var batch = yourCollection
        .Skip(i*batchSize)
        .Take(batchSize);

    // Do something with batch
}

Upvotes: 0

Foole
Foole

Reputation: 4850

This will partition the list into a list of lists of however many items you specify.

public static IEnumerable<IEnumerable<T>> Partition<T>(this IEnumerable<T> source, int size)
{
    int i = 0;
    List<T> list = new List<T>(size);
    foreach (T item in source)
    {
        list.Add(item);
        if (++i == size)
        {
            yield return list;
            list = new List<T>(size);
            i = 0;
        }
    }
    if (list.Count > 0)
        yield return list;
}

Upvotes: 3

Lee
Lee

Reputation: 144126

I don't think linq is really suitable for this sort of processing - it is mainly useful for performing operations on whole sequences rather than splitting or modifying them. I would do this by accessing the underlying IEnumerator<T> since any method using Take and Skip are going to be quite inefficient.

public static void Batch<T>(this IEnumerable<T> items, int batchSize, Action<IEnumerable<T>> batchAction)
{
    if (batchSize < 1) throw new ArgumentException();

    List<T> buffer = new List<T>();
    using (var enumerator = (items ?? Enumerable.Empty<T>()).GetEnumerator())
    {
        while (enumerator.MoveNext())
        {
            buffer.Add(enumerator.Current);
            if (buffer.Count == batchSize)
            {
                batchAction(buffer);
                buffer.Clear();
            }
        }

        //execute for remaining items
        if (buffer.Count > 0)
        {
            batchAction(buffer);
        }
    }
}

Upvotes: 2

LukeH
LukeH

Reputation: 269278

int batchSize = 100;
var batched = yourCollection.Select((x, i) => new { Val = x, Idx = i })
                            .GroupBy(x => x.Idx / batchSize,
                                     (k, g) => g.Select(x => x.Val));

// and then to demonstrate...
foreach (var batch in batched)
{
    Console.WriteLine("Processing batch...");

    foreach (var item in batch)
    {
        Console.WriteLine("Processing item: " + item);
    }
}

Upvotes: 12

Andrey
Andrey

Reputation: 60055

static void test(IEnumerable<object> objects)
{
    while (objects.Any())
    {
        foreach (object o in objects.Take(100))
        {
        }
        objects = objects.Skip(100); 
    }
}

:)

Upvotes: 12

Related Questions