CODEWITHSUNDEEP
fluentdamazon-kinesis-firehosetd-agent
Dhirendra Rawal
Dhirendra Rawal

Reputation: 1

data loss while sending from fluentd to aws kinesis firehose

We are using fluentd to send logs to aws kinesis firehose. We can see few records not sent to aws kinesis firehose every now and then. Here is our settings in fluentd.

    <system>
          log_level info
      </system>
      <source>
          @type tail
          path "/var/log/app/tracy.log*"
          pos_file "/var/tmp/tracy.log.pos"
          pos_file_compaction_interval 72h
          @log_level "error"
          tag "tracylog"
          <parse>
                @type "json"
                time_key False
          </parse>
      </source>
      <source>
         @type monitor_agent
         bind 127.0.0.1
         port 24220
      </source>
      <match tracylog>
          @type "kinesis_firehose"
          region "${awsRegion}"
          delivery_stream_name "${delivery_stream_name}"
          <instance_profile_credentials>
          </instance_profile_credentials>
          <buffer>
              # Frequency of ingestion
              flush_interval 30s
              flush_thread_count 4
              chunk_limit_size 1m
          </buffer>
      </match>

Upvotes: 0

Views: 508

Answers (1)

Dhirendra Rawal
Dhirendra Rawal

Reputation: 1

A few changes in the config fixed my issue:

  <system>
      log_level info
  </system>
  <source>
      @type tail
      path "/var/log/app/tracy.log*"
      pos_file "/var/tmp/tracy.log.pos"
      pos_file_compaction_interval 72h
      read_from_head true
      follow_inodes true
      @log_level "error"
      tag "tracylog"
      <parse>
            @type "json"
            time_key False
      </parse>
  </source>
  <source>
     @type monitor_agent
     bind 127.0.0.1
     port 24220
  </source>
  <match tracylog>
      @type "kinesis_firehose"
      region "${awsRegion}"
      delivery_stream_name "${delivery_stream_name}"

      <instance_profile_credentials>
      </instance_profile_credentials>
      <buffer>
        flush_interval 2
        flush_thread_interval 0.1
        flush_thread_burst_interval 0.01
        flush_thread_count 8
      </buffer>

Upvotes: 0

Related Questions