CODEWITHSUNDEEP
loggingelastic-stacksystemdfilebeat
Alex Guerra
Alex Guerra

Reputation: 2736

How to ingest logs from app managed by systemd

I have a service that logs structured lines of json to stdout. Using Upstart, I could add console.log to the config file and Upstart would manage saving stdout to /var/log/upstart/<service-name>.log. Another service, filebeat, would watch this log file, parsing the lines as json and then forwarding them to elasticsearch to be indexed. These logs were managed by logrotate and backed up nightly to s3.

As far as I can tell, there's no console log equivalent in systemd. The issue is then what's the best way to ship my logs out to elasticsearch?

Here are the options I've come up with:

Build log file management into the application

This is what I'm doing right now, but it feels very ugly. Maybe I'm being too dogmatic, but this isn't "12-factor" compliant (I don't want the application to worry about how its stdout is ingested), and I've already dealt with annoying debugging issues with respect to file permissions.

journalbeat

There's another unofficial "beat" called journalbeat that can ingest logs from the systemd journal. The major issue here is that I don't think it supports processing json log lines like filebeat does. I could rip that stuff out of filebeat and send a pr to journalbeat.

Shell redirection, or similar

The idea here is to start the process with a shell or some other stdout-managing program and use that for controlling how the logs are saved. I think this is pretty much a non-starter since I use [Service]Type=notify and sdnotify for startup notification (and that's not going away).

Use logstash

Logstash can ingest from journald and has all of the capabilities to parse out json and forward to es. Configuration will be more complex and I'll have to run logstash, which is heavier than filebeat, but whatever. Leaning towards this right now.

Systemd sorcery

I don't know enough about systemd to know what's possible here. I see that StandardOutput can be set to a file descriptor provided by a socket unit file, but I'm not sure how all of that fits together or if it's even possible/pragmatic.

Sorry this question was so long. Any thoughts are helpful!

Upvotes: 1

Views: 5099

Answers (1)

Mark Stosberg
Mark Stosberg

Reputation: 13381

Continuing to log to STDOUT is a good idea. It is one of the principles of the Twelve Factor App is also recommend by systemd as well as being well supported by container solutions.

Building log management into the app is a bad idea. The app should do one thing and do it well. Log management gets into thorny issues of log rotation or how to handle remote log shipping when the connection is temporarily down.

Journalbeat seems like a good idea. Have you tested that it doesn't work with JSON logs? The internal log structure of the systemd journal is very JSON-like.

If Journalbeat doesn't work, using a syslog daemon like rsyslog to handle the task. The systemd journal is often forwarded to to syslog by default anyway. rsyslog has a module for forwarding logs to Elasticsearch.

Upvotes: 2

Related Questions