CODEWITHSUNDEEP
c#.netthreadpoolout-of-memory
zSynopsis
zSynopsis

Reputation: 4854

Threadpool out of memory exception

I keep getting an out of memory exception on the following code and i was wondering if there was something i could do to stop this from happening.

  private static List<string> MyIds { get; set; }
  private static object LockObject { get; set; }
  private static int Counter { get; set; }
  private static readonly NumOfThreads = 5;

  static void Main(string[] args)
  {
      try
      {
          Console.Clear();
          LockObject = new object();
          // Pull id's into memory (A list of around 1 million ids)
          MyIds = _repository.GetIds();
          for (int i = 0; i < NumOfThreads ; i++)
                ThreadPool.QueueUserWorkItem(new WaitCallback(DoWork), (object)i);

       }
       catch (Exception ex)
       {
           Console.WriteLine(ex.StackTrace);
       }
   }

   public static void DoWork(Object stateInfo)
   {
        while (MyList.Count > 0)
        {
            lock (LockObject)
            {
                if (MyList.Count == 0)
              return;

             string id = MyList[0];

          var record = _repository.GetRecord(id);
             _repository.Add(record);

          Counter++;
          if (Counter % 100 == 0)
                            System.Console.WriteLine(DateTime.Now + " - Imported " + Counter.ToString() + " Records..");

                MyList.RemoveAt(0);
            }
        }
    }

Thanks for any help

Upvotes: 1

Views: 2726

Answers (1)

Tim Lloyd
Tim Lloyd

Reputation: 38434

You are removing from the beginning of the list which will cause a new list to be generated and the old one copied into it. This will cause havoc with the Large Object Heap given that you are dealing with lists with a large number of elements.

If you must use this type of design remove items in a reverse direction, this will prevent copying of the underlying array of the List i.e. remove from the end towards the beginning.

A better design would be to use a counter that you increment using Interlocked.Increment and use this to access members in your list. You can do this safely as you are not changing your list after you have created it.

Updated

From my comment

You are serializing access to all code in DoWork so there's no point in using multiple threads.

Something like the following will avoid the issue with removing from your ID list, plus allow you to potentially retrieve items concurrently from your repository and so utilize those extra threads. This would have to me measured though - adding threads is not a guarantee of performance improvement.

Also if your "_repository" is a collection, ensure that you size it to the same size as your ID list size. This will prevent a lot of intermediate array copying as the collection grows as you add items.

    private static int _counter = -1;

    public static void DoWork(Object stateInfo)
    {
        int index;

        while ((index = Interlocked.Increment(ref _counter)) < MyList.Count)
        {
            string id = MyList[index];

            var record = _repository.GetRecord(id);

            lock (LockObject)
            {                    
                _repository.Add(record);
            }

            if (index % 100 == 0)
                Console.WriteLine(DateTime.Now + " - Imported " + (index + 1) + " Records..");
        }
    }

Upvotes: 5

Related Questions