CODEWITHSUNDEEP
fluent-bit
Dewald
Dewald

Reputation: 71

Fluent Bit Tail Input only reads the first few lines per log file until it is restarted again

The Problem

I have a Fluent Bit service (running in a docker container) that needs to tail log files (mounted from the host into the container) and then forward those logs to Elasticsearch. For this PoC I create a new log file every minute (eg. spring-boot-logger-2021-05-25_10_53.0.log, spring-boot-logger-2021-05-25_10_54.0.log etc)

I can see that Fluent Bit picks up all the files, but it only reads and forwards the first few lines of a file (Each log entry is a single line and formated in JSON). Only when the Fluent Bit container is restarted does it read and forward the rest of the files.

To demonstrate this issue, I have a script that generates 200 log entries over a period of 100 seconds (ie. 2 logs per second). After running this script, I get a small number of entries in Elastic as shown in this image. Here one can see that there are only 72 entries with large gaps between the entries.

Once I restart the Fluent Bit container it processes the rest of the files and fill in all the logs as show in this image.

Here is my Fluent Bit config file:

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug
    Parsers_File   /fluent-bit/etc/parsers.conf

[INPUT]
    Name  tail
    Parser docker
    Path  /var/log/serviceA/*.log
    Tag   service.A
    DB    /var/db/ServiceA
    Refresh_Interval 30

[INPUT]
    Name  tail
    Parser docker
    Path  /var/log/serviceB/*.log
    Tag   service.B
    DB    /var/db/ServiceB
    Refresh_Interval 30

[OUTPUT]
    Name  stdout
    Match service.*

[OUTPUT]
    Name  es
    Host  es01
    Port  9200
    Logstash_Format On
    tls   Off
    Match service.*

What I've tried I've tried the following:

I've also used the debug versions of these containers to confirm that the files mounted correctly into the container and that they reflect all the logs (when Fluent Bit does not pick it up)

Fluent Bit's log level has also been set to debug, but there are no hints or errors in the logs.

Has anybody else experienced this issue?

Upvotes: 7

Views: 7937

Answers (2)

Jon Palle Hansen
Jon Palle Hansen

Reputation: 11

I have had the same issue with fluent-bit on Openshift using glusterfs for persistent volumes.

My workaround has been to fork the official repo and build a new fluent-bit Docker image after making a small addition to the Dockerfile:

RUN cmake ... \
    ... \
    -DFLB_INOTIFY=Off \
    ..

However, in the meantime, I see that there is now a configuration parameter called Inotify_Watcher in the tail input documentation, which I guess can be used for exactly this.

Upvotes: 1

Taner
Taner

Reputation: 4569

I am not sure but I think Fluentbit (1.7 and 1.8) has bug(s) to access on shared logs in PV. It has rights, sees files but not fetches the log lines after its first fetch.

I found the solution by placed the Fluentbit as a sidecar, not a seperated pod.

Upvotes: 0

Related Questions