CODEWITHSUNDEEP
tcpipsyslogfluentd
Haihung
Haihung

Reputation: 21

Unable to capture syslog client IP addresses using Fluentd @tcp <source> parameter 'source_address_key' with warning <source> is not used

Need help with fluentd config to allow capturing of syslog client IP addresses in the record.

I'm using docker container to start a fluentd instance to aggregate logs from remote syslog clients to be send to Elasticsearch for indexing.

Managed to get the setup working with below @type syslog . 

<source>
  @type syslog
  tag syslog
  protocol_type tcp
  port 5140
  frame_type octet_count
  <parse>
    @type syslog
    message_format rfc5424
    with_priority true
    rfc5424_time_format %Y-%m-%dT%H:%M:%S+%z
  </parse>
</source>

However as there are many syslog clients to be aggregated, intend to differentiate the logs by adding the IP address of the syslog clients to the record.

Hence tried to use @type tcp and the source_address_key parameter listed in the documentation to capture the incoming client IP address.

"The field name for the client's IP address. If you set this option, Fluentd automatically adds the remote address to each data record."

<source>
  @type tcp
  tag tcp.events
  port 5140
  source_address_key client_addr
   <parse>
     @type regexp
     expression /^[\w :]+\<(?<pri>[0-9]{1,3})\>[1-9]\d{0,2} (?<time>[^ ]+) (?<host>[^ ]+)[- ]+(?<message>[\[\w\:\]\- \;\=\'\,\(\.\)\#]+)$/
   </parse>
</source>

The output of terminal shows the following warning message:

[warn]: parameter 'source_address_key' in <source>
  @type tcp
  tag "tcp.events"
  port 5140
  source_address_key client_addr
  <parse>
    @type "regexp"
    expression /^[\w :]+\<(?<pri>[0-9]{1,3})\>[1-9]\d{0,2} (?<time>[^ ]+) (?<host>[^ ]+)[- ]+(?<message>[\[\w\:\]\- \;\=\'\,\(\.\)\#]+)$/
  </parse>
</source> is not used.

And from the output, only managed to capture the syslog messages, but no IP address added to the record.

Upvotes: 1

Views: 1502

Answers (1)

Haihung
Haihung

Reputation: 21

Found the issue. Seems like source_address_key doesn't work with @type tcp.

Found that it only works in @type syslog.

Posting back here in case anyone encounter the same issue. 

    # get logs from syslog
<source>
  @type syslog
  tag syslog
  protocol_type tcp
  source_address_key client_addr
  port 5140
  frame_type octet_count
  <parse>
    @type syslog
    message_format rfc5424
    with_priority true
    rfc5424_time_format %Y-%m-%dT%H:%M:%S+%z
  </parse>
</source>

Upvotes: 1

Related Questions