CODEWITHSUNDEEP
loggingsyslog
Jono
Jono

Reputation: 1750

Best practices for network resilient centralized logging

We have 50+ servers all generating lots of logs and they are sent via rsyslog to a third party (Logtrust) so we can do centralized analysis. Logtrust allows for real time analysis as well.

As of now we do not keep any local file logs of our applications. They all write to syslog - which writes directly to Logtrust.

I dont want to keep all our logs locally, but I am concerned about the hard third party/off machine dependency. If for some reason our logs dont make it to them (local config issues, local relays down, third party server down, networking/ISP issues), we have not visibility into our logs.

We can have rsyslog also write to a local file and log rotate it so it does not get too big (1Gb for example), or we can have each application manage its own tee to syslog and a local file.

How big would this file cap be? Should we log locally at all? What are industry best practices here?

Upvotes: 1

Views: 716

Answers (1)

Stephane Martin
Stephane Martin

Reputation: 1642

There are multiple possibilities. rsyslog is a very stable and performant product, so in any cases I would recommend to always make your applications talk to rsyslog, and not write directly to some file.

In rsyslog you can define multiple actions for your log messages.

One action is responsible to send logs to Logtrust. That action should have a disk-assisted memory queue (http://www.rsyslog.com/doc/v8-stable/concepts/queues.html). When rsyslog can't relay some messages to Logtrust, the messages will be buffered in the rsyslog queue until delivery is possible.

You can define more actions:

  • you can ask rsyslog to also write each message to a local file with a file action. Then use logrotate to purge older files. The retention period should be big enough to give you time to investigate in case of incidents. So typically between 3 days (cope with weekends) and 1 week.

  • if you can afford the complexity, you can set up a centralized rsyslog instance, in-house. The centralized server would gather the logs from all your applications, and serve as a backup solution in case Logtrust shuts down. Between the local rsyslog instance and the centralized one, use the protocol RELP to ensure no logs get lost.

  • If you need to be able to search efficiently the logs in your centralized instance, the current best practice is to store further them into Elasticsearch or Graylog (which is also Elasticsearch based). Rsyslog can natively forward logs to both products.

Upvotes: 1

Related Questions