td-agent-bit delivers only after restart and then stops

Question

In my Office I am forced to work still with td-agent-bit, an upgrade to the recent fluent-bit should be possible later this year.

I have trouble with the following configuration:

[INPUT]
    Name tail
    Path /ws/int/xxx/log/c/log/lvasc-xxx/import-management.log,/ws/int/xxx/log/c/log/lvasc-xxx/importer.log
    Path_Key On
    Tag ax_imports_c
    Parser ax_parser
    DB /home/pgostack/tail_c.db
    Buffer_Max_Size 250MB
    Refresh_Interval 10
[OUTPUT]
    Name opensearch
    Match ax_imports_c
    Host lvast-pgopbos01
    Port 9200
    Index imports_index
    Type  imports_type
    HTTP_User cccc
    HTTP_Passwd cccccc!
    tls On
    tls.verify Off
    Suppress_Type_Name On
    Logstash_Format On
    Logstash_Prefix ax_imports_c

When I start it works like a charm and sends all the log-data to OpenSearch - but then stopps. Only when I restart td-agent-bit it sends the data (filling the gap) to OS. I tried all the parameters I can think of, but still the same. Another conf with an exec-routine works without flaws...

Help & hints would be very welcome!

Greetings from Germany

Thomas (bearbeitet)

Answer 1

I would look into the Buffer_Max_Size value, which could be too high for the volume of logs your data source produces. In fact, the data won't be sent until the buffer is full or until the agent is restarted, whatever happens first