Reputation: 101192
I've made a log parser which parses log files containing a lot of rows. The current implementation uses a LinkedList<T>
to build a list of entries and is quite fast.
I've also built a log viewer which uses a virtual list view. As you might understand it do not work very well with a linked list.
I'm thinking about using the LinkedList
when doing the initial parsing and then allocate a List
when done (with capacity specified). Then I simply use AddRange
to add the log entries to the list.
More new items will be added later on since I use a FileSystemWatcher
on the log file, but not in the same rate as the initial parsing.
Is it a good idea to switch or do you have any better suggestions?
The parsing is done by something that implements the following interface (one parser per log format).
Public Interface ILogParser
Sub Parse(ByVal stream As IO.Stream)
Sub Parse(ByVal stream As IO.Stream, ByVal offset As Long)
Event EntryParsed(ByVal sender As Object, ByVal e As ParsedEntryEventArgs)
Event Completed(ByVal sender As Object, ByVal e As EventArgs)
End Interface
The logviewer subscribes on both events and adds each entry from the EntryParsed
event to the LinkedList
. The Completed
event is triggered when the whole log (file)stream have been parsed.
When completed I start keeping track of the last position in the stream that was successfully parsed and the log parser method Parse(fileStream, lastPosition)
is called each time a FileSystemWatcher
event is triggered.
Upvotes: 0
Views: 122
Reputation: 101192
I've switched from List<T>
to LinkedList<T>
when I parse the log file for the first time. When the first parsing is done I copy everything to a List<T>
to be able to use it in my virtual list view.
The solution works great and the performance gain is a plus ;)
Upvotes: 0
Reputation: 5623
I have written alot of parsers, and i use List<> instead of LinkedList, what i normaly do is that i make a guess on the size of the List before i start parsing ( You can often guess from the size of the file or line count how big your List needs to be).
That being said, if using List makes you system have alot of OurOfMemory execptions i would swap to LinkedList<>.
Side note, when i say lots of lines i mean 200k +
Upvotes: 1
Reputation: 3827
LinkedList does not provide any benefit here since you do not have any requirement to "remove" or "insert" log entries. If logs are tiemstamp base, these would be "appended".
Upvotes: 0