CODEWITHSUNDEEP
c#datasettraceout-of-memory
Guillaume V
Guillaume V

Reputation: 1081

Memory consumption of a large dataset

I have a .net winform c# program. I collect traces on my program, with my own trace system. The trace show information about what the user did and other usefull information. I store each trace in a dataset. An example, I add a row in the dataset every second. The programe dispose the traces only when the program encounter a undhandler exception or when the program close.

The problem is : The dataset begin to be very large in the memory, and if the program is running enough time, a out of memory exception occur.

I search the web and i found some ideas. - put the dataset in a file on hard drive ( in this case I must open a big file each time i want to add a trace) - paging the dataset

I must find a way to free the memory use by this dataset by keeping all the information it containts and also keep a way to add additionnal rows to this dataset.

Someone have ideas or guidance ? Thank you

Upvotes: 2

Views: 1818

Answers (3)

Yahia
Yahia

Reputation: 70379

You could save the content of the dataset to disk (every hour in its own file), clear the in-memory data, run for an hour in-memory, save the data into a separate file and so on.

This way your memory consumption should be ok and your performance gets only hit once an hour.

Since the files on disk all have the same structure they can be merged whenever you need the "complete data".

Although I would recommend using some existing logging library - there are lots of well-tested one out there (free and commercial).

Upvotes: 3

Venki
Venki

Reputation: 1459

Use log4net before you use it for your Winforms Project create a config like this preferably in your App.config file.

<configuration>
  <configsections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net">
  </section>
  <log4net>
    <root>
      <level value="DEBUG">
      <appender-ref ref="LogFileAppender">
    </appender-ref>
    <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender">
      <param name="File" value="C:\MyLogFolder\Customlog.txt"><param name="AppendToFile" value="true"><rollingstyle value="Size">
      <maxsizerollbackups value="10">
      <maximumfilesize value="10MB">
      <staticlogfilename value="true">
      <layout type="log4net.Layout.PatternLayout">
        <param name="ConversionPattern" value="%-5p%d{yyyy-MM-dd hh:mm:ss} – %m%n"></layout>
    </staticlogfilename>
  </maximumfilesize>
</maxsizerollbackups>
</rollingstyle></appender></level></root></log4net></configsection

Its very simple to understand. You can see the Path to the log file listed in the App.config configuration (you can customize other stuff pertaining to your application from the config settings). Then from your application use this to initialize the log4net

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using log4net;
using log4net.Config;

namespace Log4Net
{
    public partial class Form1 : Form
    {

        private static readonly ILog log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);  
        public Form1()
        {
            InitializeComponent();
            log4net.Config.XmlConfigurator.Configure();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            log.Warn("Custom Warning Message");
            log.Debug("Custom Debug Message");
            log.Info("Custom Info Message");
            log.Error("Custom Error Message");
        }


    }
}

That should be just about it. Its simple and very powerful.

Upvotes: 2

kprobst
kprobst

Reputation: 16651

You could use something like NLog or the logging block in the Enterprise Library instead of a dataset, and use a rolling file target to write your trace. .NET itself also has tracing facilities, although they're less advanced than NLog and other projects.

Generally speaking, it's always a bad idea to hold diagnostic data in memory, since a crash would wipe it out. A Dataset is intended for data access and quick read/write operations, not this sort of thing. Obviously it has a much higher memory overhead than a simple file logger.

Upvotes: 2

Related Questions