CODEWITHSUNDEEP
fluentdtd-agent
Expert wanna be
Expert wanna be

Reputation: 10624

FluentD, how to grep only spcific logs

2019/08/13 13:13:17 [DEBUG] Hello, world!
2019/08/13 13:13:17 [INFO] Ignore me
2019/08/13 13:13:17 [INFO] SPECIFIC_LOG :{"name": "mark"}

I have like above logs, and I need to grep only the logs which contains 'SPECIFIC_LOG' and I want to ignore the others.

I tried to set the config like this,

<source>
    @type tail
    path ./sample.log
    tag debug.sample
    <parse>
        @type regexp
        expression /\[\w+\] SPECIFIC_LOG\s:(?<message>.*)$/
    </parse>
</source>

<filter debug.**>
    @type parser
    key_name message
    format json
</filter>

And it is working for the matched log with pattern, but for the not matched log, I got warning which says 

#0 pattern not matched: "2019/08/13 13:13:17 [DEBUG] Hello, world!"

How can I grep only the log which is matched the pattern, so that I can resolve the warning?

Upvotes: 1

Views: 2298

Answers (1)

Imran
Imran

Reputation: 6245

It is just a warning that pattern did not match and in my opinion, it can be ignored.

To ignore such warnings, you can set emit_invalid_record_to_error false option.

<filter debug.**>
    @type parser
    key_name message
    format json
    emit_invalid_record_to_error false
</filter>

More info on this flag - https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error

Earlier versions had suppress_parse_error_log flag and now it has been replaced with emit_invalid_record_to_error.

suppress_parse_error_log is missing. What are the alternatives?

Since v1, parser filter doesn't support suppress_parse_error_log parameter because parser filter uses @ERROR feature instead of internal logging to rescue invalid records. If you want to simply ignore invalid records, set emit_invalid_record_to_error false. See also emit_invalid_record_to_error parameter.

Upvotes: 0

Related Questions