CODEWITHSUNDEEP
kubernetesefk
Arpan Sharma
Arpan Sharma

Reputation: 415

EFK Kubernetes Log Aggregation

I am installing EFK stack to 100 days old cluster. Fluentd will pick up the logs of 100 days and will start sending to Elastic. Is there a provision that fluentd starts aggregating the log from today and not from beginning life cycle of the cluster?

Upvotes: 0

Views: 281

Answers (1)

aga
aga

Reputation: 3893

If you look into Fluentd documentation, you can find limit_recently_modified flag, which allows limit the watching files that the modification time is within the specified time range.

Here's how the limit_recently_modified can be used in conf file:

...
<source>
    exclude_path ["/var/log/wedge/*/*/MattDaemon*.log"]
    path_key source
    format none
    read_from_head true
    tag foo.*
    path /var/log/wedge/*/*/*.log
    pos_file /var/log/td-agent/wedgelog
    limit_recently_modified 86400s
    @type tail
</source>
...

Another option is to use Filebeat (instead of Fluentd), where you can find ignore_older flag. Filebeat ignores any files that were modified before the specified timespan.

I hope it will helps you.

Upvotes: 1

Related Questions