CODEWITHSUNDEEP
elasticsearcherlanglogstash
ihi
ihi

Reputation: 107

Erlang disk_log : how to output raw text/JSON for interfacing with logstash?

I have :

  1. a cluster with lots of Erlang VMs dumping logs with disk_log
  2. logstash will aggregate the logs written by disk_log
  3. elasticsearch is used together with logstash to search through the logs in (semi)real-time.

My problem is that disk_log's output is in Erlang's internal format. How do I interface to logstash ? Could I parse the disk_log output file in another Erlang VM/process while it is still being written (it's a log, after all, it will be open as long as the Erlang VM is running) and dump it into raw text/JSON for logstash ? Any problems with that ?

Upvotes: 3

Views: 757

Answers (1)

tkowal
tkowal

Reputation: 9289

You have couple of options:

  1. You should be able to read the file with another Erlang VM, while it is still open using disk_log:chunk/2 and disk_log:chunk/3. Then, you can translate the terms to JSON (for example using Jiffy or JSX) and send them to logstash (for example via UDP).

This solutions will be slow, because it needs first to write to the disk and then read from it. disk_log is usually used for its high performance, so adding this much overhead doesn't feel right.

  1. You can add sending the logs to graylog via UDP directly in your application. This is great, because it has much less overhead on the disk, but you need to remember to add both types of logs (disk_log and JSON via UDP) in every place.

  2. Forget the disk_log and use lager. Lager became standard, when you need to log anything in Erlang. The feature list is impressing. You can define different backends. In your case: file and graylog. File backend is included with lager and you can find graylog backend here. Even with all that power, it still has high performance.

Upvotes: 2

Related Questions