Why logs appear to be parsed correctly by Fluent-Bit but don't show up in Opensearch-Dashboards?

Question

Stack

Opensearch + Opensearch-Dashboards + Fluent-Bit

Goal

Parse Apache Tomcat's catalina.out log file

Configs

fluent-bit.conf:

[SERVICE]
    Flush                 5
    Log_Level             info
    Parsers_File          parsers.conf


[INPUT]
    Name   tail
    Tag   tomcat.access
    Path    /u01/tomcat/apache-tomcat/logs/access*.txt
    Parser   tomcat_access

[INPUT]
    Name tail
    Tag tomcat.catalina
    Path /u01/tomcat/apache-tomcat/logs/catalina.out

[FILTER]
    Name multiline
    Match tomcat.catalina
    multiline.key_content log
    multiline.parser tomcat_catalina_multiline, java

[FILTER]
    Name     parser
    Match    tomcat.catalina
    Key_Name log
    Parser   tomcat_catalina_spring
    Preserve_key On

[OUTPUT]
    Name file
    Match tomcat.catalina
    Path /u01/fluent-bit
    Format out_file

[OUTPUT]
    Name                  opensearch
    Match                 tomcat*
    Host                  <host>
    Port                  <port>
    Suppress_Type_Name    On
    Index                 tomcat-test

parsers.conf:

[PARSER]
    Name        tomcat_access
    Format      regex
    Regex       ^(?<host>\S+) (?<ident>\S+) (?<user>\S+) \[(?<time>.*)\] "(?<method>\S+) (?<url>\S+) (?<protocol>\S+)" (?<status>\S+) (?<size>\S+)$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        tomcat_catalina_spring
    Format      regex
    Regex       ^(?<time>\d+-\d+-\d+ \d+:\d+:\d+\.\d+)\s+(?<level>\S+) \d+ --- \[\s*(?<thread>[^\]]+)\] (?<context>\S+)\s+: (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L

[MULTILINE_PARSER]
    name          tomcat_catalina_multiline
    type          regex
    flush_timeout 500
    #
    # rules |   state name    | regex pattern                             | next state
    # ------|-----------------|--------------------------------------------------------
    rule      "start_state"      "/^(\d+-\d+-\d+ \d+:\d+:\d+\.\d+)(.*)$/"  "empty_row"
    rule      "empty_row"        "/^$/"                                    "error_row"
    rule      "error_row"        "/^.*$/"                                  "stacktrace"
    rule      "stacktrace"       "/^(\s*at .*|)$/"                         "stacktrace"

File OUTPUT section

I've configured an output to a file for debugging purposes. The tomcat.catalina contents seem to be just fine:

tomcat.catalina: [1723676238.680000000, {"level":"ERROR","thread":"main","context":"com.zaxxer.hikari.pool.HikariPool","message":"HikariPool-1 - Exception during pool initialization."}]
tomcat.catalina: [1723665438.703756094, {"log":"java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection\n\tat ]
... <enormous multiline log entry here ...>"]
tomcat.catalina: [1723665438.703751094, {"log":"\n"}]
tomcat.catalina: [1723676238.682000000, {"level":"WARN","thread":"main","context":"ConfigServletWebServerApplicationContext","message":"Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException... <long message here>"}]
### ... ###
### some seem to be parsed exactly how I need entries, same as following... ###
tomcat.catalina: [1723665438.703843277, {"log":"\n"}]
tomcat.catalina: [1723676238.694000000, {"level":"INFO","thread":"main","context":"ConditionEvaluationReportLoggingListener"}]
tomcat.catalina: [1723665438.703843277, {"log":"\n"}]
tomcat.catalina: [1723676238.723000000, {"level":"ERROR","thread":"main","context":"o.s.boot.SpringApplication","message":"Application run failed"}]
### and so on... ###

The problem

When I go to my Opensearch-Dashboards Discover tab, the parsed single line entries (those with level, thread, context and message fields) are not there. Only multiline ones and these:

tomcat.catalina: [1723665438.703843277, {"log":"\n"}]

(The least show up as empty log ofc, the \n symbol emerges only in JSON view.)

The fields are present at tomcat-test index Mappings and also show up in Fields of tomcat-test index pattern.

I've refreshed the tomcat-test index pattern fields, deleted the tomcat-test index, restarted both Fluent-Bit and Tomcat to recreate the index and push some new entries there. No errors reported by either Opensearch-Dashboards or Fluent-Bit.

!UPD!

The logs are there but... in the future. 3 hours ahead. Can also be found in Dashboards by adjusting the time filter.

E.g. if compare those two:

tomcat.catalina: [1723665438.703843277, {"log":"\n"}]
tomcat.catalina: [1723676238.694000000, {"level":"INFO","thread":"main","context":"ConditionEvaluationReportLoggingListener"}]

Apologies for not looking at the log entries' timestamps carefully.

What am I actually missing here?

Answer 1

I'm in UTC+3 timezone. Time_Offset parser parameter does the magic (https://github.com/fluent/fluent-bit/issues/326).

Still have no idea why exactly these logs are "in the future" though.