CODEWITHSUNDEEP
c#filesystemsfifo
Sabre
Sabre

Reputation: 2400

FIFO log file in C#

I am making an evolution for a log file system I have had in place for a few builds of a service I develop on. I had previously been opening the file, appending data, and prior to writing checking to see if the log file had grown over a predetermined size, if so starting a new log.

So say the log size was 100mb, at that size I delete, and start a new file, but I loose history, functional, but not the best model.

What I want to do is a FIFO model that would chop off the top and add to the end while keeping it consistently no larger than 100mb, and at least as far back as that represents.

The data is high speed in a failure prone industrial environment, so keeping it all in memory and writing the whole file at interval has proven unreliable. (SSD, fast enough to do it reasonably most of the time, spinners fail too often to tolerate)

Likewise the records are of greatly variable length (formatted as XML nodes, so parsing them back out accounts for this easily)

So the only workable model I have come up with thus far is to keep smaller slices (say 10mb) chunks, create new ones then delete the oldest 10mb slice on count >= 10.

What I would prefer to do is be able to keep the file on disk and work with the tag ends.

Open to suggestions on how this might be best achieved in a reasonable manner, or is there no reasonable manner and the layered multi log approach will be the best option?

Upvotes: 0

Views: 1123

Answers (2)

Kelmen
Kelmen

Reputation: 1063

i dunno why u need this feature "chop off the top and add to the end while keeping it consistently no larger than 100mb".

general design approach is archiving. simply rename the oversized file to another file, or move it to somwhere else, then using back the same filename as new file.

simple as this is.

Upvotes: 1

Corey
Corey

Reputation: 16574

The biggest issue with expiring old log entries in a single file is that you have to rewrite the file's content in order to expire older entries. This isn't too bad for small files (up to a few MB in size), but once you get to the point where rewriting takes a significant period of time it becomes problematic.

One of the more common ways to retire logs is to rename the existing log file and/or start a new file. Lots of programs do it that way, with either dated log file names or by using a sequential numbering system - logfile, logfile.1, logfile.2, etc. with higher-numbered files being older. You can add compression to the process to further reduce the storage requirements for expired files, etc.

Another option is to use a more database-like format, or an out-and-out database like SQLite to store your log entries. The primary downside of this of course is that your log files become more difficult to read, since they're not just in plain text form. It's simple enough to write a dump-to-text program whose output can be piped to a log parser... but even this will probably require a change in the way your consumers are interfacing with the log file.

The problem as stated is unlikely to be realistically solvable, I suspect. On the one hand you have the limitations of file manipulation, and on the other the fact that your log consumers are many and varied and therefore changes to the logging structure will be an involved process.

About all I can suggest is that you trial a log aging process similar to this:

  • Rename current log file
  • Walk renamed file and copy desired contents to new log file
  • Discard or archive renamed log

Beware duplication or data loss.

Upvotes: 3

Related Questions