CODEWITHSUNDEEP
loggingamazon-web-servicesamazon-s3fluentd
Petar Zivkovic
Petar Zivkovic

Reputation: 1010

Fluentd copy output plugin not sending same data to both destinations?

I have a web server stack with multiple nodes (auto scaling group), and each web server is configured to use Fluentd to forward log files to a central collector which saves the logs in an S3 bucket.

I am testing a new collector, and configured the web servers to send a copy of an access log to both the old and the new collector, but there is a huge discrepancy in the aggregated files.

The old collector is saving 2x the log data. I can't figure out for the life of me what the issue is, unless the data is getting lost somewhere on the way to the new collector (and how could I best test this?).

Web server td-agent.conf

<source>
  type tail
  path /var/log/some/access.log
  tag some-access.log
  format tsv
  keys time,userId,position,userAgent,platform
  pos_file /tmp/fluent.some-access.log.pos
  rotate_wait
</source>

<match some-access.log>
  type copy
  <store>
    type forward
    <server>
      host log-collector-1.lan
      port 24224
    </server>
  </store>
  <store>
    type forward
    <server>
      host log-collector-2.lan
      port 24224
    </server>
  </store>
</match>

The log collectors have the exact same config... so no need to look there.

Upvotes: 2

Views: 3088

Answers (1)

repeatedly
repeatedly

Reputation: 718

I first heard such behaviour. I assume log-collector-1.lan refers same server as log-collector-2.lan.

You didn't paste aggregator side configuration so just a guess. How to check the 'saving 2x the log data'?

You can check fluentd receives logs or not by following way.

Upvotes: 2

Related Questions