CODEWITHSUNDEEP
logginglogstashapache-kafkadistributedfluentd
Evhz
Evhz

Reputation: 9266

Which server logging solution is less processor consuming among Kafka, Fluentd and Logstash?

I have a distributed logging system to monitor load-balanced server entities. It is basic for me, that the server does not invest a lot of processor time in the logging process, allowing the application to run with the maximum resources possible.

It would be nice to know, which of those alternatives is "cheaper" in terms of processor time or, in case, to be recommended any other solution for this matter.

Upvotes: 0

Views: 931

Answers (4)

Marko Bonaci
Marko Bonaci

Reputation: 5706

You should probably use just a simple lightweight rsyslog, syslog-ng or syslogd.

Depends on which technologies you'd like to use?

Upvotes: 1

Marina
Marina

Reputation: 4064

We are using Flume to collect logs and ship to Kafka - it is very low on CPU usage, and memory-wise- it is up to you how much buffering you want to do. You can also write your own Flume interceptor if you need custom parsing/routing of data.

Upvotes: 1

Kiyoto Tamura
Kiyoto Tamura

Reputation: 730

First of all, Kafka is not a log collector. It's a distributed message queue and can work with log collectors like Fluentd and Logstash both as consumers and producers.

Instead of opinions, let's put some numbers.

  1. CPU Usage: This depends on how much filtering and data processing you do on the client-side using Fluentd and/or Logstash. If you do minimal processing, both can process 10,000+ messages per second. Both can take advantage of multiple CPUs (http://docs.fluentd.org/articles/in_multiprocess, for example)
  2. Memory: Fluentd uses about 40MB of memory and Logstash uses about 100MB of memory. If this is too much, Logstash has Beats and Fluentd as Fluentd Forwarders (https://github.com/fluent/fluentd-forwarder).

Upvotes: 0

Teddy Ma
Teddy Ma

Reputation: 1136

fluentd and logstash all have processor cost, better to not run them on your application server. For, kafka, since it is only its client be running on your server, it should be cheaper, but then the stability of your server depends on the stability of the kafka servers.

A better solution might be to log to files and install a cheaper log collector/forwarder to forward the log files to another server for running any log parsers with.

Upvotes: -1

Related Questions