CODEWITHSUNDEEP
loggingarchitecturesoftware-design
Grim
Grim

Reputation: 2097

What are ways to solve the logging haystack problem in larger systems?

I have always thought that the more logging you have the better for debugging/troubleshooting. It just makes it so much easier given you are logging the correct things and not logging noobish things. I have always done this on the smaller scale. How would you do this in larger systems? Would it be worth it, or would it be more beneficial to just log what is needed to identify there is a problem and what the problem is but not very specific details.

Logging haystack problem being, so many log lines/rows that its harder to track the full request or multiple requests. No separation in requests, just line/row after each other.

On the small scale I have done the following successfully and I do like the approach. See below, just a quick high level design... Yes I am a .net developer :).

FYI: Not exactly sure the best place for this question.

Upvotes: 1

Views: 280

Answers (1)

Lukas Horak
Lukas Horak

Reputation: 91

The challange how to debug large scale system is pretty broad so it's better to break it down to several separated pieces. You

Collect logs from all the systems

One of the common way is to use existing systems for centralized log collection and managment. You can use commerical solution like Splunk or open source stack like ELK (Elastic Logstash Kibana). These systems have usually main 3 parts

  1. Log collection - listen to the log outputs and forward them to the centralized location. It can be done in several different ways depending on the needs and used technology.
  2. Centralized storage - Some DBMS system, which allows to store large number of data and index them so they can be searched efficiently
  3. GUI - To allow users to search the DBMS and setup dashboards and alerts

https://www.elastic.co/what-is/elk-stack

Track the requests across multiple sysems

The best approach for most setups is to have generate unique correlationId, which is then carried across all the systems and used as part of the log message.

https://dzone.com/articles/correlation-id-for-logging-in-microservices

Logger configuration

For the debugging the most important is to actually generate the log messages. This topic is fairly complex on it's own. As too much logging make the code not very clear to read, too much logging reduce the performance, incomprehensible log messages makes it hard to understand the problem.

Upvotes: 1

Related Questions