CODEWITHSUNDEEP
filtergrepfluent-bit
Sudeep Amin
Sudeep Amin

Reputation: 101

Exclude pattern on a grep filter on fluent-bit does not seem to be working

I am trying to filter out a few records from the tail input to fluent-bit. But that does not seem to work. From the log files I need to exclude from all records with key value 'log' 1) Records that have 1 or more digits followed by a space 2) records with value 'Series' anywhere on the line 3) records with the value 'transacttime' anywhere on the line.

They could be the same or different records

[INPUT]
    Name         tail
    Path         /mnt/volume_nyc3_03/xenfix*.out
    Tag          genfix
    DB           /mnt/volume_nyc3_03/ggantel-gf.db


[FILTER]
    Name         grep
    Match        *
    Exclude      log ^[0-9]*\
    Exclude      log *Series*
    Exclude      log *transacttime*

[OUTPUT]
    Name         pulsar
    Match        *
    Host         somerandom-id.us-east-1.elb.amazonaws.com
    Port         6650
    Topic        persistent://public/default/genfixlogs

[OUTPUT]
    Name         stdout
    Match        genfix

This does not exclude any records from the output as seen below

{"log":"0 1"}
----- got message -----
{"log":"2019-09-17 21:25:08.636465 Series([], Name: transacttime, dtype: datetime64[ns])"}
----- got message -----
{"log":"2019-09-17 21:25:08.633038 Series([], Name: transacttime, dtype: datetime64[ns])"}
----- got message -----
{"log":"2019-09-17 21:25:08.680237 Series([], Name: transacttime, dtype: datetime64[ns])"}
----- got message -----
{"log":"2019-09-17 21:25:08.890903 Series([], Name: transacttime, dtype: datetime64[ns])"}

Upvotes: 6

Views: 13379

Answers (1)

Yuki Iwamoto
Yuki Iwamoto

Reputation: 141

You need to enclose in // when using regex.

Below is an example.

[FILTER]
    Name         grep
    Match        *
    Exclude      log /^[0-9]*/
    Exclude      log /.*Series.*/
    Exclude      log /.*transacttime.*/

Upvotes: 14

Related Questions