CODEWITHSUNDEEP
c#performancelinq
Gary
Gary

Reputation: 732

Populate a large list of objects with details from another list

I have a large database query that returns around 100k records into an in-memory list. I need to link a list of related employees to each record (also around 100k records), but I'm struggling to get useable performance.

foreach (var detail in reportData.Details)
{
    detail.Employees = employees
      .Where(x => x.AccountingDocumentItemId == detail.AccountingDocumentItemId)
      .Select(x => x.Employee)
      .ToList();

    detail.Employee = String.Join(", ", detail.Employees);
}

The above code takes over 8 minutes to complete. I've narrowed down the speed issue to the first line in the for loop where it finds the related employees. If I leave out the ToList() it's super fast, but then the next line immediately causes the issues where the String.Join causes the Where to execute.

I'm obviously approaching this from the wrong angle, but I've exhausted the options I think would work.

Upvotes: 2

Views: 295

Answers (1)

Dmitrii Bychenko
Dmitrii Bychenko

Reputation: 186803

You current code has O(n ** 2) time complexity (nested loops) and thus you have 1e5 * 1e5 ~ 1e10 (10 billions) operations to perform which takes 8 minutes to complete.

Let's extract a dictionary in order to have O(n) time complexity (~1e5 operations only):

var dict = reportData
  .Details
  .GroupBy(item => item.AccountingDocumentItemId,
           item => item.Employee)
  .ToDictionary(chunk => chunk.Key, 
                chunk => chunk.ToList());

foreach (var detail in reportData.Details) {
  detail.Employees = dict.TryGetValue(detail.AccountingDocumentItemId, out var list) 
    ? list.ToList()        // copy of the list 
    : new List<MyClass>(); // put the right type instead of MyType

  detail.Employee = String.Join(", ", detail.Employees);
}

Upvotes: 6

Related Questions