CODEWITHSUNDEEP
nosql
Hitesh
Hitesh

Reputation: 1077

NoSQL as local storage for logging and tracing

We are developing application which will have many physical servers. We want to use NoSQL for logging and tracing since it does not required structured data.

We don't want to have Centralized logging.

Can we install NoSQL (any one) in each server and store logging/tracing details? Will NoSQL impact my actually process in the server? Is it good idea to do it?

Upvotes: 2

Views: 1724

Answers (2)

Kazuki Ohta
Kazuki Ohta

Reputation: 1441

Problem1: Data Collection

Many people're using NoSQL solutions for storing application logs. The first challenge you may have is how to collect huge amount of data from various data sources reliably with ease of management. One concern of not having log collection layer, is lock contention of database caused by high write throughput.

So basically having log collection layer is recommended. There're some open-source log collector implementation such as syslog, Fluentd, Scribe, and Flume :)

Problem2: Storage & Processing

The next big problem is how to store and process data. The backend infrastructure requires a lot of changes as the data volume increase. At first, you can use MongoDB to store all of your data, but at some moment you end up using Apache Hadoop to architect a massively scalable architecture.

Here's an example architecture of having Fluentd for log collection, and MongoDB for log storage and processing.

fluentd plus mongodb

Here're some links to put the Apache Logs into Amazon S3, MongoDB, or Hadoop HDFS by Fluentd.

Disclaimer: I'm a committer of Fluentd project.

Upvotes: 5

Yogesh Prajapati
Yogesh Prajapati

Reputation: 4870

definitely this is good idea for doing same thing with nosql rather than sql. because in logging and tracing volume of data is high and ratio of retrieving data is also high.

you for logging and tracing you need complex reports for analysis so nosql is better for you.

also nosql support distributed environment so you create infrastructure at different geographic location.

Upvotes: 0

Related Questions