CODEWITHSUNDEEP
loggingtext-mining
Vladimir Ralev
Vladimir Ralev

Reputation: 1381

Tools for reducing large logs files

I work with huge log files - 1GB or so that have many user sessions in it while I only care about one session.

I can usually narrow-down the general area of the file that covers the session i am interested in just by searching for session id (takes 2+ minutes). After that I want to remove the data before and after the events that occurred in the user session to make my subsequent searches faster (because I have narrowed down the area of interest now).

I like to load huge log files in google chrome and use the search highlight feature which displays the area of interest with markers on the scrollbar, but it doesn't really work on files bigger than 200MB and doesn't allow me to remove irrelevant parts of the logs to make subsequent searches faster.

I imagine it is a common problem. It would be a huge time-saver if I can find such tool.

Thanks.

Upvotes: 0

Views: 70

Answers (2)

Victor Welling
Victor Welling

Reputation: 1897

Splunk is a nice tool for log monitoring and analysis. Perhaps it covers a bit more ground than you might need, but it's definitely worth taking a look at. There's a free license available, which is limited to 500MB/day and an enterprise license if you want to go all out (license comparison table).

Upvotes: 1

Jens Schauder
Jens Schauder

Reputation: 81990

many unix command line tools help with this kind of stuff. Especially grep allows to find lines containing some string or pattern (session id). By default it returns the row, but I can also return n rows before or after.

Upvotes: 1

Related Questions