CODEWITHSUNDEEP
mongodbpostgresqlhadooploggingnosql
denizeren
denizeren

Reputation: 934

Log viewing utility database choice

I will be implementing log viewing utility soon. But I stuck with DB choice. My requirements are like below:

I know that PostgreSQL will work if I fragment tables. But will I able to get this performance written above. As I understood NoSQL is better choice for log storing, since logs are not very structured. I saw an example like below and it seems promising using hadoop-hbase-lucene: http://blog.mgm-tp.com/2010/03/hadoop-log-management-part1/

But before deciding I wanted to ask if anybody did a choice like this before and could give me an idea. Which DBMS will fit this task best?

Upvotes: 5

Views: 164

Answers (2)

mys
mys

Reputation: 2473

My logs are very structured :)

I would say you don't need database you need search engine:

  • Solr based on Lucene and it packages everything what you need together
  • ElasticSearch another Lucene based search engine
  • Sphinx nice thing is that you can use multiple sources per search index -- enrich your raw logs with other events
  • Scribe Facebook way to search and collect logs

Update for @JustBob: Most of the mentioned solutions can work with flat file w/o affecting performance. All of then need inverted index which is the hardest part to build or maintain. You can update index in batch mode or on-line. Index can be stored in RDBMS, NoSQL, or custom "flat file" storage format (custom - maintained by search engine application)

Upvotes: 5

slownage
slownage

Reputation: 154

You can find a lot of information here:

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

See which fits your needs.

Anyway for such a task NoSQL is the right choice.

You should also consider the learning curve, MongoDB / CouchDB, even though they don't perform such as Cassandra or Hadoop, they are easier to learn.

MongoDB being used by Craigslist to store old archives: http://www.10gen.com/presentations/mongodb-craigslist-one-year-later

Upvotes: 4

Related Questions