Using StreamWriter to implement a rolling log, and deleting from top

Question

My C# winforms 4.0 application has been using a thread-safe streamwriter to do internal, debug logging information. When my app opens, it deletes the file, and recreates it. When the app closes, it saves the file.

What I'd like to do is modify my application so that it does appending instead of replacing. This is a simple fix.

However, here's my question:

I'd like to keep my log file AROUND 10 megabytes maximum. My constraint would be simple. When you go to close the file, if the file is greater than 10 megabytes, trim out the first 10%.

Is there a 'better' way then doing the following:

1. Close the file
2. Check if the file is > 10 meg
3. If so, open the file
4. Parse the entire thing
5. Cull the first 10%
6. Write the file back out
7. Close

Edit: well, I ended up rolling my own (shown following) the suggestion to move overt to Log4Net is a good one, but the time it woudl take to learn the new library and move all my log statements (thousands) over isn't time effective for the small enhancement I was trying to make.

  private static void PerformFileTrim(string filename)
  {
     var FileSize = Convert.ToDecimal((new System.IO.FileInfo(filename)).Length);

     if (FileSize > 5000000)
     {
        var file = File.ReadAllLines(filename).ToList();
        var AmountToCull = (int)(file.Count * 0.33); 
        var trimmed = file.Skip(AmountToCull).ToList();
        File.WriteAllLines(filename, trimmed);
     }
  }

Answer 1

I was looking through the win32 api, and I'm not even sure it's possible to do this with native win32 vfs calls, nevermind through .Net.

About the only solution I would have would be to use memory-mapped files and move the data manually, which .Net seems to support as of .Net 4.0.

Memory Mapped Files

Answer 2

I liked greggorob64's solution but also wanted to zip the old file. This has everything you need other than the part of compressing the old file to a zip, which you can find here: Create zip file in memory from bytes (text with arbitrary encoding)

static int iMaxLogLength = 2000; // Probably should be bigger, say 200,000
static int KeepLines = 5; // minimum of how much of the old log to leave

public static void ManageLogs(string strFileName)
{
    try
    {
        FileInfo fi = new FileInfo(strFileName);
        if (fi.Length > iMaxLogLength) // if the log file length is already too long
        {
            int TotalLines = 0;
                var file = File.ReadAllLines(strFileName);
                var LineArray = file.ToList();
                var AmountToCull = (int)(LineArray.Count - KeepLines);
                var trimmed = LineArray.Skip(AmountToCull).ToList();
                File.WriteAllLines(strFileName, trimmed);
                string archiveName = strFileName + "-" + DateTime.Now.ToString("MM-dd-yyyy") + ".zip";
                File.WriteAllBytes(archiveName, Compression.Zip(string.Join("\n", file)));
        }

    }
    catch (Exception ex)
    {
        Console.WriteLine("Failed to write to logfile : " + ex.Message);
    }
}

I have this as part of the initialization / reinitialization section of my application, so it gets run a few times a day.

ErrorLogging.ManageLogs("Application.log");

Answer 3

This function will allow you to rotate your log based on weekdays. First time y our application will launch on Monday, will check for any existing entry for Monday Date, if not already initialized for today will discard old entries and reinitialize new file. Onwards for whole of that day, file will keep appending the text to same log file.

So, total 7 log files will be created. debug-Mon.txt, debog-Tue.txt...

it will also add the method name which actually logged the message along with date time. very useful for general purpose use.

private void log(string text)
        {
            string dd = DateTime.Now.ToString("yyyy-MM-dd");
            string mm = DateTime.Now.ToString("ddd");

            if (File.Exists("debug-" + mm + ".txt"))
            {
                String contents = File.ReadAllText("debug-" + mm + ".txt");


                if (!contents.Contains("Date: " + dd))
                {
                    File.Delete("debug-" + mm + ".txt");
                }
            }

            File.AppendAllText("debug-" + mm + ".txt", "\r\nDate: " + DateTime.Now.ToString("yyyy-MM-dd HH:mm:s") + " =>\t" + new System.Diagnostics.StackFrame(1, true).GetMethod().Name + "\t" + text);

        }

Answer 4

The solutions here did not really work for me. I took user3902302's answer, which again was based on bigtech's answer and wrote a complete class. Also, I am NOT using StreamWriter, you can change the one line (AppendAllText against the StreamWrite aequivalent).

There is little error handling (e. g. re-try access when it is failing, though the lock should catch all internal concurrent access).

This might be enough for some people who had to use a big solution like log4net or nlog before. (And log4net RollingAppender is not even thread-safe, this one is. :) )

public class RollingLogger
{
    readonly static string LOG_FILE = @"c:\temp\logfile.log";
    readonly static int MaxRolledLogCount = 3;
    readonly static int MaxLogSize = 1024; // 1 * 1024 * 1024; <- small value for testing that it works, you can try yourself, and then use a reasonable size, like 1M-10M

    public static void LogMessage(string msg)
    {
        lock (LOG_FILE) // lock is optional, but.. should this ever be called by multiple threads, it is safer
        {
            RollLogFile(LOG_FILE);
            File.AppendAllText(LOG_FILE, msg + Environment.NewLine, Encoding.UTF8);
        }
    }

    private static void RollLogFile(string logFilePath)
    {
        try
        {
            var length = new FileInfo(logFilePath).Length;

            if (length > MaxLogSize)
            {
                var path = Path.GetDirectoryName(logFilePath);
                var wildLogName = Path.GetFileNameWithoutExtension(logFilePath) + "*" + Path.GetExtension(logFilePath);
                var bareLogFilePath = Path.Combine(path, Path.GetFileNameWithoutExtension(logFilePath));
                string[] logFileList = Directory.GetFiles(path, wildLogName, SearchOption.TopDirectoryOnly);
                if (logFileList.Length > 0)
                {
                    // only take files like logfilename.log and logfilename.0.log, so there also can be a maximum of 10 additional rolled files (0..9)
                    var rolledLogFileList = logFileList.Where(fileName => fileName.Length == (logFilePath.Length + 2)).ToArray();
                    Array.Sort(rolledLogFileList, 0, rolledLogFileList.Length);
                    if (rolledLogFileList.Length >= MaxRolledLogCount)
                    {
                        File.Delete(rolledLogFileList[MaxRolledLogCount - 1]);
                        var list = rolledLogFileList.ToList();
                        list.RemoveAt(MaxRolledLogCount - 1);
                        rolledLogFileList = list.ToArray();
                    }
                    // move remaining rolled files
                    for (int i = rolledLogFileList.Length; i > 0; --i)
                        File.Move(rolledLogFileList[i - 1], bareLogFilePath + "." + i + Path.GetExtension(logFilePath));
                    var targetPath = bareLogFilePath + ".0" + Path.GetExtension(logFilePath);
                    // move original file
                    File.Move(logFilePath, targetPath);
                }
            }
        }
        catch (Exception ex)
        {
            System.Diagnostics.Debug.WriteLine(ex.ToString());
        }
    }
}

edit:
Since I just noticed that you asked a slightly different question: should your lines vary greatly in size, this would be a variation (, that in 90% of cases does not improve over yours, though, and might be very slightly faster, also introduced a new unhandled error (\n not being present)):

    private static void PerformFileTrim(string filename)
    {
        var fileSize = (new System.IO.FileInfo(filename)).Length;

        if (fileSize > 5000000)
        {
            var text = File.ReadAllText(filename);
            var amountToCull = (int)(text.Length * 0.33);
            amountToCull = text.IndexOf('\n', amountToCull);
            var trimmedText = text.Substring(amountToCull + 1);
            File.WriteAllText(filename, trimmedText);
        }
    }

Answer 5

This is derived from bigtech's answer:

private static string RollLogFile()
    {
        string path = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
        string appName = Path.GetFileNameWithoutExtension(Environment.GetCommandLineArgs()[0]);
        string wildLogName = string.Format("{0}*.log",appName);

        int fileCounter = 0;
        string[] logFileList = Directory.GetFiles(path, wildLogName, SearchOption.TopDirectoryOnly);
        if (logFileList.Length > 0)
        {
            Array.Sort(logFileList, 0, logFileList.Length);
            fileCounter = logFileList.Length - 1;
            //Make sure we apply the MaxLogCount (but only once to reduce the delay) 
            if (logFileList.Length > MaxLogCount)
            {
                //Too many files - remove one and rename the others
                File.Delete(logFileList[0]);
                for (int i = 1; i < logFileList.Length; i++)
                {
                    File.Move(logFileList[i], logFileList[i - 1]); 
                }
                --fileCounter;
            }

            string currFilePath = logFileList[fileCounter];
            FileInfo f = new FileInfo(currFilePath);
            if (f.Length < MaxLogSize)
            {
                //still room in the current file
                return currFilePath;
            }
            else
            {
                //need another filename
                ++fileCounter;                  
            }

        }
        return string.Format("{0}{1}{2}{3:00}.log", path, Path.DirectorySeparatorChar, appName, fileCounter);
    }

Usage:

string logFileName = RollLogFile();
using (StreamWriter sw = new StreamWriter(logFileName, true))
{
    sw.AutoFlush = true;
    sw.WriteLine(string.Format("{0:u} {1}", DateTime.Now, message));
}

Answer 6

I researched this once and never came up with anything, but I can offer you plan B here:

I use the selection below to keep a maximum of 3 log files. At first, log file 1 is created and appended to. When it exceeds maxsize, log 2 and later log 3 are created. When log 3 is too large, log 1 is deleted and the remaining logs get pushed down the stack.

string[] logFileList = Directory.GetFiles(Path.GetTempPath(), "add_all_*.log", SearchOption.TopDirectoryOnly);
if (logFileList.Count() > 1)
{
    Array.Sort(logFileList, 0, logFileList.Count());
}

if (logFileList.Any())
{
    string currFilePath = logFileList.Last();
    string[] dotSplit = currFilePath.Split('.');
    string lastChars = dotSplit[0].Substring(dotSplit[0].Length - 3);
    ctr = Int32.Parse(lastChars);
    FileInfo f = new FileInfo(currFilePath);

    if (f.Length > MaxLogSize)
    {
        if (logFileList.Count() > MaxLogCount)
        {
            File.Delete(logFileList[0]);
            for (int i = 1; i < MaxLogCount + 1; i++)
            {
                Debug.WriteLine(string.Format("moving: {0} {1}", logFileList[i], logFileList[i - 1]));
                File.Move(logFileList[i], logFileList[i - 1]); // push older log files back, in order to pop new log on top
            }
        }
        else
        {
            ctr++;
        }
    }
}