CODEWITHSUNDEEP
filtergrepfluentdtd-agent
user3675572
user3675572

Reputation: 23

Fluentd exclude filter works and doesn't work

I'm looking into fluentd to send Apache logs to an http output. I'm partly successful but I don't understand the grep filter it seems. Some things I put in there work and others don't, I don't really see a difference so I'm stumped..

Example logline:

10.50.1.36 - - [31/Mar/2020:12:48:26 +0000] "GET /index.php?r=Hle/Create&ReturnUrl=/index.php?r=Hle/Admin HTTP/1.1" 200 34291 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"

My conf file filter looks like this:

<filter apache.access>
  @type grep
  <exclude>
    key path
    pattern /Create/
  </exclude>
</filter>

My input gets tagged with the same apache.access. The example above works but this doesn't:

<filter apache.access>
  @type grep
  <exclude>
    key method
    pattern /GET/
  </exclude>
</filter>

I would expect this to work just as well? The incoming logfile is parsed as apache and this is the output:

{"host":"10.50.1.36","method":"GET","path":"/index.php?r=Hle/Create&ReturnUrl=/index.php?r=Hle/Admin","code":200,"size":34291,"agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36","time":"31/Mar/2020:12:48:26 +0000"}

So the method field definitely holds the GET value but it doesn't get filtered out for some reason? Why do some things work and others don't? I also tried with other options in field path but so far it even seems that only string that hold a capital get filtered. For example, if the path were to contain the word 'create' I'm not able to exlude it with pattern /create/ ? On the other hand, if it contains Create I am able to filter it out with pattern /Create/ ?

Just in case the entire conf file, maybe that explains it?

    ## read apache logs continuously and tags apache.access
     <source>
      @type tail
      @id input_tail
      <parse>
        @type apache2
        time_key "time"
        keep_time_key true
      </parse>
      path /var/log/apache2/httpd
      pos_file /var/log/td-agent/httpd-access.log.pos
      tag apache.access
    </source>


    <filter apache.access>
      @type grep
      <exclude>
        key method
        pattern /GET/
      </exclude>
    </filter>

    <filter apache.access>
      @type record_transformer
      <record>
        hostname "#{Socket.gethostname}"
      </record>
      remove_keys referer,user
    </filter>

<match apache.*>
  @type http

  endpoint http://localhost:9000
  open_timeout 2

  <format>
    @type json
  </format>
  <buffer>
    flush_interval 2s
  </buffer>
</match>

Upvotes: 0

Views: 1359

Answers (1)

jj40308
jj40308

Reputation: 21

The config you provide works fine to me. But the logline :

10.50.1.36 - - [31/Mar/2020:12:48:26 +0000] "GET /index.php?r=Hle/Create&ReturnUrl=/index.php?r=Hle/Admin HTTP/1.1" 200 34291 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"

seems like missing the referer column, according to doc

Upvotes: 0

Related Questions