CODEWITHSUNDEEP
apacheloggingfluentd
nixmind
nixmind

Reputation: 2266

Strange "pattern not match" error on fluentd

Can someone tell if its normal that fluentd raise this error in td-agent.log file?

2015-07-31 13:15:19 +0000 [warn]: pattern not match: "- - - [31/Jul/2015:13:15:19 +0000] GET http://172.31.108.218/ HTTP/1.1 200 0 \"-\" \"ELB-HealthChecker/1.0\""

While this is a well formated apache2 log:

- - - [31/Jul/2015:13:15:19 +0000] GET http://172.31.108.218/ HTTP/1.1 200 0 \"-\" \"ELB-HealthChecker/1.0\"

And here is the source configuration:

<source>
  type tail
  format apache2
  path /var/log/varnish/varnishncsa.log
  pos_file /var/log/td-agent/tmp/access.log.pos
  tag "apache2.varnish-access"
</source>

I can't figure out what's wrong there above.

Upvotes: 4

Views: 16776

Answers (2)

nixmind
nixmind

Reputation: 2266

The problem is that these ELB-HealthChecker line log has an empty referer ip field. And then the log doesn't match apache2 log format for fluentd.

So the way to fix that is to filter logs with ELB-HealthChecker user-agent.

Upvotes: 0

geerlingguy
geerlingguy

Reputation: 4802

Instead of finding some way to filter out logs from ELB-HealthChecker, you can set your own format for the Apache access log that is a little more flexible in terms of the first couple fields. I ran into this same error when getting /server-status checks from collectd (using it to monitor for SignalFx).

Setting the source like so:

<source>
  type tail
  format /^(?<host>[^ ]*(?:\s+[^ ]+)*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
  time_format %d/%b/%Y:%H:%M:%S %z
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.pos
  tag apache2.log
</source>

Allows both log lines like:

172.18.0.2:80 127.0.0.1 - - [08/Aug/2017:19:58:38 +0000] "GET /server-status?auto HTTP/1.1" 200 508 "-" "collectd/5.7.2.sfx0"

As well as:

192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"

You can test format regex matching using Fluentular.

See related: Fluentd apache log format with multiple host ip

Upvotes: 0

Related Questions