CODEWITHSUNDEEP
c++algorithmfiletext
Janz Dott
Janz Dott

Reputation: 325

Algorithm for writing limited number of lines to text file

I have a program where I need to write text lines to a log file very frequently. I would like to limit the number of lines in the log file to 1000. When I write lines to the file, it should append them normally. Once the file reaches 1000 lines, I'd like to get rid of the first line and then append the new one. Does anyone know if there is a way to do this without rewriting the entire file each time?

Upvotes: 5

Views: 1777

Answers (5)

Scott Mermelstein
Scott Mermelstein

Reputation: 15397

Any time you want to log, you can open the file, read your write index, jump to the position, and write the fixed-width log entry. When your index hits your upper threshold, simply set it back to 0.

There are a lot of warnings with this, though - first is that each proper log entry (assuming you close the file in between) will require an open, a read, a seek, a write, a seek, a write and a close - to find your index, go to it, write the new entry, then update your index. You also have the inherent issues of writing a fixed-size data element. Also, a human reader will depend on your content to know where the "beginning" of the file is. Most people expect "line 1" to be the first line.

I'm a much bigger advocate for simply having a few files and "rolling" them, so that each file on its own is coherent, but if you want just one file with a fixed number of lines, the circular buffer idea can work.

Upvotes: 0

villekulla
villekulla

Reputation: 1085

When you only want to use one file, and the length of the lines are not constant, there is no way without rewriting the whole file.

Depending on how often you are appending to the file, I don't see any problem doing so. 1000 lines of approx 100 chars are only approx 100kb, which is not to much. Additionally you may add some hysteresis.

However:

  • If the line length is constant (or you hard-limit the line length to some constant), you could just overwrite the oldest line. But then you have to keep track of the log file positions of old/new lines
  • I would use two files: The first one where you append lines. When the file gets full, rename it to a second one, and fill the first one from the beginning.

Upvotes: 0

Roman
Roman

Reputation: 1867

I think you still have to scan the file to find out how many lines are in the file at this moment. In that case, you can put it in some sort of buffer that you could easily add and delete from. Then you do your logging and when you are done, you could "re-write" the file with the buffer (or only last 1000 lines). Other alternatives are discussed above. And yeah, try to avoid deleting line-by-line. Generally, it is a costly operation.

I've found some similar topics here and on CodeProject:

Small logger class;

Flexible logger class using standard streams in C++

http://www.codeproject.com/Articles/584794/Simple-logger-for-Cplusplus

Hope you find them useful :)

Upvotes: 1

Ivan Ishchenko
Ivan Ishchenko

Reputation: 1526

All files have to be aligned to FS cluster size. So, no, there's no way. You can append a line to a file, but you can't delete the first line without file rewriting. You can use 2 files by turns. Or use some buffer in memory and flush it periodically.

Upvotes: 1

Nick Raverty
Nick Raverty

Reputation: 431

Generally it's a little bit better for a case like this to remove more than one line at a time from the beginning.

That is, if your limit is 1000 lines, and you hit 1000 lines, delete the first 300 or so, and then resume writing. That way, you're not performing the delete operation with every single line written thereafter, only every 300 times. If you need to persist 1000 lines, then instead keep up to 1300 and delete 300 when 1300 is reached.

Upvotes: 1

Related Questions