CODEWITHSUNDEEP
javafilelogging
user2071270
user2071270

Reputation: 172

Parsing Tomcat Log files

Hello Everyone, I have been trying to this from a week or so. But not figure out a way. I work with tomcats and my client regularly send me log files in 2 to 3GB's stating there was a issue of file not found etc.. Some times they dont have the proper information to grep through the log files. SO I decided to build a tool that can parse all log files and can categorize the logs accordingly. Now i cannot store 4 GB of data in memory and I cannot put it back into the file because reading 4GB will take a lot of time. Even though I am using file channels and threads. Data Base is certainly not a option since it will again slow the system down. So I want to know is there is any other way to store the parsed contents so that whenever i want to check 404 error i must get all 404 errors in a list.

I do not wish to use a database. So database is certainly not a answer for this.

Upvotes: 0

Views: 2487

Answers (2)

Maclean Pinto
Maclean Pinto

Reputation: 1135

You can use Apache Lucene. Use nio file handlers for divining the file into chucks and use Apache Lucene for indexing and text searching. This might not solve your complete problem but is a better solution if you do not wish to use data base.

Upvotes: 0

Mikkel Løkke
Mikkel Løkke

Reputation: 3749

It doesn't matter whether "You want to use a database" or not. What you're doing is essentially building a graph of data. This is what databases are designed for. Now you can choose to use one that someone else wrote, and is widely tested, or you can choose to roll your own. Either way you're using a database, whether you want to or not.

If you want a lightweight, embeddable, well performing, document/graph "No SQL" database that works well with Maven, OrientDB is your friend, and using it is very intuitive. Plus you can choose whether you want to use an in-memory database, a file backed database, or a more traditional client/server solution, depending on your needs. Best part is that it has an Object abstraction layer, so you don't even have to mess about with an ORM framework.

You really should try it. It'll make all your pains go away.

Linky: http://www.orientdb.org/

Upvotes: 2

Related Questions