C# memory usage for creating objects in a for loop

Question

I have a complex database conversion console app that reads from an old database, does a bunch of things, and puts into the new database.

I'm having an escalating memory problem where my mem usage (as monitored in task manager) constantly climbs and eventually slows down the process to a halt.

I've boiled it down to the simplest possible test POC to try and understand what's going on.

for (int i = 0; i < 100000; i++)
{
  TestObj testc = new TestObj
    {
      myTest = "testing asdf"
    };
}
public class TestObj
{
    public string myTest;
}

My thought was that each testc that is created in the loop wouldn't survive past the end of the iteration, but the way the memory is tracking it seems like the application is holding on to every instance of testc.

I've done a good amount of research and experimentation but I feel like there is something I'm missing here. Shouldn't I be able to run this and have memory utilization stay rather constant?

Answer 1

My guess is that the memory your seeing being eaten up is not managed memory. Here's the key issue: the GC only knows about managed memory. If you are dealing with other objects that have a 'footprint' outside of the managed world - eg. files, COM objects, database connections, windows, and so on - those will take up memory within that process, but because the GC is only aware of the managed portion of their footprint, the unmanaged portion can grow and grow without the GC ever being aware that a collection is needed.

Or, put another way, GC is great for managing pure memory, but lousy at managing resources (files, COM objects, HANDLEs, windows, etc) - if you are using those intensively, then you may need to actively close / dispose or otherwise clean them up as soon as you are done with them, and not rely on GC.

The example you give with the object+string is pure managed, there's no resources here, so likely it will hit some upper limit, collection will kick in, and it will level off, but not make the system slow down (at least not too much!).

What objects are you actually creating/using within the loop? If they are wrappers to external resources, such as database connections or similar, check if they implement IDispose and then use .Dispose or the using() pattern, or see if there's a Close/Disconnect or other method to release the resource.

Answer 2

You probably need to understand that C# uses "garbage collection" to manage the lifetime of objects.

Your loop doesn't give the garbage collector the opportunity to actually dispose of the objects, so they are hanging in memory until the compiler has determined the most opportune moment to dispose of them.

UPDATE
I agree with Sunny that garbage collection is most likely NOT your issue. The only way increased memory usage could impact the execution speed of your program is if you have reached a point where the machine is swapping lots of RAM to disk.

I would suggest that you need to profile your database interaction instead. At what point does the program start slowing down? Is the disk queue length on the database server growing? How many queries is it trying to execute at once?

There are two possibilities here. The first is that the slow down is occuring pulling large amounts of data across the wire. The second is that the slow down is occuring pushing large amounts of data across the wire.

Whichever it is, look at the particular server involved. It might not be able to support the response time you are asking of it.

Answer 3

Garbage collection is not deterministic, so you cannot expect to see the results immediatelly, unless you explicitly call GC.Collect(). Next, if you are working with ADO.NET, you need to make sure that you are disposing every object which implements IDisposable. Not doing this will unnecessarily delay collection.

Also, make sure that your simplified example really mimics what's going on in your actual code. For example, if your objects attach their methods as event handlers for some other long living object, they will be kept alive by this reference as long as this single object lives.

To check the actual status of your memory, a list of all living objects at certain point, and actual references which keep them alive, an irreplaceable tool is WinDbg with SoS extensions.

With your example, the way it is, it is impossible that garbage collector doesn't clean up eventually.

1. You are not creating new String instances or anything but your test objects.
   Due to string interning in .NET, only one string constant exists in your application, and your test object only gets its reference (not a copy).

2. Your object only takes 16 bytes on the heap (in a 32-bit environment)
   A minimum of 12 bytes, plus a single string reference, makes a 16 byte large object. That's pretty small, even if there is a million of them.

The loop creates a bunch of small, short lived objects which don't create any new references, and don't attach any event handlers. GC is great at collecting these kind of objects, but collects them only when it feels necessary.

Answer 4

You're missing one key thing about garbage collection: The GC won't run until it needs to. So yes, even though it's nice enough to clean up after you, it still won't do it until more memory is needed.

Answer 5

Each iteration of the loop will create a new object in memory the sizeof(TestObj). The end result is that when you leave the for loop the memory usage will be sizeof(TestObj) * 100000; Since your objects don't ever go out of scope (according to your code), they will won't be ever GCed.

If you are concerned about the memory, you could put the TestObj in a using statement (using (obj = new TestObj()) if your test object implements IDisposible. You could also try a multithreaded approach if your processing would work in multitheaded manner, and spawn a worker thread each iteration. The TestObj will probably be collected around the same time that the the thread is collected, it could also speed up the overall time it takes the app to do all the work.

Answer 6

Life will be a little easier if you use a struct instead...

for (int i = 0; i < 100000; i++)
{
    TestObj testc = new TestObj
    {
        myTest = "testing asdf"
    };
}

public struct TestObj
{
    public string myTest;
}

This still requires the allocation of the string but the struct won't survive. It depends on what your class really looks like, if you have a lot of value types in it this will help immensely. If you have a bunch of string/reference values your still in trouble.

Otherwise you can do something like the following:

for (int i = 0; i < 100000; i++)
{
    // do your work...

    // then every 1k cycles, see if we have > 100mb allocated
    // and force the GC to free the memory
    if(i % 1000 == 0 && GC.GetTotalMemory(false) > 100000000)
        GC.Collect();
}

Note: This is an ugly 'hacky' sort of thing to do; however, sometimes it's the quickest solution to the problem.

Update

Addtionally you need to make sure you are not hitting the LOH (Large object heap) as this can be a source of memory contention. As a general rule keep strings, byte[], ect, under 85kb. This means that strings need to me less than 42k characters in length.

Answer 7

You can use using anywere it is possible, (for any type implementing IDisposable interface)

using(var conn = new SqlConnection ())
{
}

Do not keep SqlConnection open longer than it really needs, use WeakReference class for internal memory representation alongside with temporary serialization/deserealization storage. Use data caching, mediator pattern, observer pattern

Answer 8

I would just try that :

TestObj testc;
for (int i = 0; i < 100000; i++)
{
  testc = new TestObj
    {
      myTest = "testing asdf"
    };
}

Or like Artur said, a using statement if your class is IDisposable.

Answer 9

The first thing I'd try is to separate the code that goes inside the loop from the loop code by adding a method.

public void DoProcessing()
{
    for (int i = 0; i < 100000; i++)
    {
        ProcessItem();
    }
}

private void ProcessItem()
{
    TestObj testc = new TestObj
    {
        myTest = "testing asdf"
    };
}

public class TestObj
{
    public string myTest;
}

Answer 10

Your test objects will be cleaned away when generation 1 of living objects would be full. Or when someone calls GC.Collect(). Try to call it and you'll see that nothing will grow up.

GC doesn't clean memory immediately after it doesn't referenced by anyone in .NET. And your TestObject is very-very shy in memory consuming as there's only one shared interned instance of your string in the heap. Thus you could have created many of them and GC doesn't interfere. They are just too small to make a difference.