CODEWITHSUNDEEP
c++memory-leaksofstream
fwg
fwg

Reputation: 1038

ofstream leaking memory

I have a C++ class that writes its data out to a binary std::ofstream. The class is storing the data as a boost:shared_array but I have eliminated this as the problem. The problem is with the call to write() on the ofstream.

The issue is that it seems to leak memory. The system it is running on is CentOS 64bit, GCC 4.1.2.

When looking at top and free when the application is running the executable itself does not continue to consume memory (as backed up by the memory profiler in Netbeans), but the amount of free system memory does decrease over time. What's more, when the application exits this memory is not reclaimed!

This is a particular issue because the intention is to write out to disk continuously at around 50MB/s for hours on end. However, once we get down to around 90MB of free system memory it seems to "stabilize" and not reduce any further and the application continues to run OK. It does however screw up the system for the other running processes, which is bad, mmkay.

Below is an ever-so slightly simplified version of the class that is causing the grief.

class WritableMessage
{
public:
    WritableMessage();
    void write(std::ofstream* const idxStream, std::ofstream* const dataStream);

private:
    IdxRecord m_idx;
    boost::shared_array<char> m_data;
};

The ofstreams are initialised and destructed elsewhere but essentially they remain open for writing "forever".

void WritableMessage::write(std::ofstream* const idxStream, std::ofstream* const dataStream)
{
    //Call the IdxRecord to write itself (just a call to idxStream->write())
    m_idx.write(idxStream, dataStream->tellp());

    //This is the main issue, because this data can be up to about 2MB in size
    //for each write.  Commenting out this line removes all issues with the memory leak
    dataStream->write(m_data.get(), m_idx.getMessageSize());

    //I'd expect the flush to clear any buffers that the ofstream maintains,
    //but apparently not
    idxStream->flush();
    dataStream->flush();
}

Upvotes: 1

Views: 3735

Answers (2)

sehe
sehe

Reputation: 393457

Use vmstat (man page)

vmstat -S m 1

It will show you cache and buffer memory. Note that buffer memory is volatile and will be freed automatically as soon as an application requests it.

I can easily show the effect on my 8GB desktop (linux) by logging in, and merely doing 'dd if=/dev/sda of=/dev/null'; the buffer memory will steadily consume all of available memory.

This is by design

Some pertinent links:

Upvotes: 3

Vitor
Vitor

Reputation: 2792

It looks like you don't have a problem, and that is just the system cache working as intended. Linux is very agressive on caching, so it will use almost all your free memory for this, but whenever an application needs more memory, it will release some of that memory and grant it to the application.

Upvotes: 4

Related Questions