CODEWITHSUNDEEP
node.jsdatabase
t8101
t8101

Reputation: 395

Best efficient way to store logs

I was thinking to store my node app's log in a database in an efficient way for future study. Can anyone suggest me easy to learn database that I can store my logs in it? I am expecting more than a thousand queries per hour. I am a bit concerned to use Mongo cause it's 16mb limit ber doc.

Is there any easy-to-learn database with drivers for node app?

Upvotes: 0

Views: 4438

Answers (2)

Peter Short
Peter Short

Reputation: 772

Hello again :) This is an answer to the question you asked in my first answer:

 Thanks. But I think they are more "search engine" rather than database...right 🤔 ?

Yes for sure, they are more a search and indexing solution rather than a simple database. And theres a good reason for it!

Imagine you implement your table for logs in a relational store like postgres. You have a simple insert to the table for each log. Now your table has a set schema, which makes it hard to get at your logs without reading all the data and processing it. Unless your schema fits the information structure of your logs. But then what if in future you have a library that wants to log its own structure of data to the table. You could have a field called "log_data" and maybe use some real text search feature... Its not ideal.

You could go for a object store solution, like you alluded to, and you will then run into other problems. You will need to solve a lot of them yourself. One of which is log staleness. Logs usually have an expiration date, they usually stop being of value after a certain period of time goes by. As your system runs, it just adds data, locking to do expensive writes, probably blocking processing on your processing node, for something that should be quick. Also, if something is locked, or something goes wrong, your app might crash the request, because of logging....

Using a dedicated logging store solves these issues. Firstly they can dynamically handle structured logs, so you can start logging new structures of information to them and they can handle it. You can efficiently query these structures, because they are effectively a search engine on that log data.

They also have the ability to roll indexes, meaning they dont get slower and slower as more and more data comes in, because they can intelligently partition the indexes by day and get rid of ones that are too old to be useful any more, solving the "I have too much logs" problem.

The apis to log are also designed to not be disruptive to the processing of your app. Logging is a nice to have, but unlike auditing, it should not take down your app if it stops functioning. Loggers usually use some async lightweight dispatch mechanism to achieve this, which you would get for free.

There is a little bit of work to set them up, but if you want to use docker and docker compose, that work is minimal compared to the benefit you will get for using the right tool for the job :)

Upvotes: 1

Peter Short
Peter Short

Reputation: 772

I would rather suggest that you look at something like elastic stack for logs. Using a database makes your logs opaque, unless you use some full text search feature of the DB. In short, log information is a different class of information to your application data, it has different persistance requirements, performance requirements etc. A relational, or non relational data store isn't the ideal place to keep log data.

Instead there are products like Elastic Search, Solr, InfluxDb etc that are designed to quickly store log info, process it and make it searchable. These products usually have a UI partner that allows you to make visualizations on top of your log data, so you can start to see the big picture. They also bring to the table he concept of "structured logging" which allows you to define some semantic data in your logs, making them even more useful.

There are probably plenty of structured loggers available for nodejs (such as winston) that integrate to elastic search and provide you with a ready API that will allow you to do powerful searches and aggregations on you log data. You could run elastic stack either hosted or via docker :

Elastic stack docker guide

Node js structured logging with winston to Elastic

Upvotes: 2

Related Questions