CODEWITHSUNDEEP
searchindexinglucenesolrlog4j
Arun Manivannan
Arun Manivannan

Reputation: 4313

log4j Log Indexing using Solr

We are finding it very hard to monitor the logs spread over a cluster of four managed servers. So, I am trying to build a simple log4j appender which uses solrj api to store the logs in the solr server. The idea is to use leverage REST of solr to build a better GUI which could help us

  1. search the logs and the display the previous and the next 50 lines or so and
  2. tail the logs

Being awful on front ends, I am trying to cookup something with GWT (a prototype version). I am planning to host the project on googlecode under ASL.

Greatly appreciate if you could throw some insights on

  1. Whether it makes sense to create a project like this ?
  2. Is using Solr for this an overkill?
  3. Any suggestions on web framework/tool which will help me build a tab-based front end for tailing.

Upvotes: 4

Views: 3423

Answers (3)

unixrules
unixrules

Reputation: 608

You can use a combination of logstash (for shipping and filtering logs) + elasticsearch (for indexing and storage) + kibana (for a pretty GUI).

Upvotes: 3

Joseph Holsten
Joseph Holsten

Reputation: 20928

The loggly folks have also built logstash, which can be backed by quite a few things, including lucene via elastic search. It can forward to graylog also.

Upvotes: 2

Eric Pugh
Eric Pugh

Reputation: 1573

Totally doable thing. Many folks have done the roll your own. A couple of useful links.. there is an online service, www.loggly.com that does this. They are actually based on Solr as the core storage engine! Obviously they have built a proprietary interface.

Another option is http://www.graylog2.org/. It is opensource. Not backed by Solr, but still very cool!

Upvotes: 0

Related Questions