CODEWITHSUNDEEP
fluentdtd-agent
Manish
Manish

Reputation: 21

fluentd: Not able to send syslogs to Elasticsearch using fluent-plugin-elasticsearch

I am setting up log forwarding for my Linux & Windows machines on a RHEL8 VM using Fluentd and Elasticsearch. This is what I have installed:

I am able to redirect the logs to the td-agent.log file but not able to forward the logs to Elasticsearch.

After installing the fluent-plugin-elasticsearch plugin, when I turn on the td-agent, it continuously generates these logs:

2023-06-14 04:45:26 -0500 [info]: #0 init worker0 logger path="/var/log/td-agent/td-agent.log" rotate_age=nil rotate_size=nil
2023-06-14 04:45:26 -0500 [info]: adding match pattern="td.." type="tdlog"
2023-06-14 04:45:26 -0500 [warn]: #0 [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="debug." type="stdout"
2023-06-14 04:45:26 -0500 [info]: adding match pattern="system." type="elasticsearch"
2023-06-14 04:45:26 -0500 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="EOFError (EOFError)"
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:324:in rescue in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/base.rb:285:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/transport/http/faraday.rb:36:in perform_request' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elastic-transport-8.2.1/lib/elastic/transport/client.rb:176:in perform_request'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-8.7.0/lib/elasticsearch.rb:71:in method_missing' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/elasticsearch-api-8.7.0/lib/elasticsearch/api/actions/info.rb:41:in info'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:498:in detect_es_major_version' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:489:in block in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/elasticsearch_index_template.rb:56:in retry_operate' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:486:in handle_last_seen_es_major_version'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.3.0/lib/fluent/plugin/out_elasticsearch.rb:338:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/plugin.rb:187:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:132:in add_match' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:74:in block in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in each' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/agent.rb:64:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/root_agent.rb:149:in configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:105:in configure'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/engine.rb:80:in run_configure' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:616:in block in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:962:in main_process' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/supervisor.rb:608:in run_worker'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.16.1/bin/fluentd:15:in <top (required)>'
2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in load' 2023-06-14 04:45:26 -0500 [error]: #0 /opt/td-agent/bin/fluentd:23:in 
2023-06-14 04:45:26 -0500 [error]: Worker 0 exited unexpectedly with status 1

This is my td-agent.conf configuration for syslog forwarding:

<source>
  @type syslog
  port 5140
  tag system
</source>

<match system.**>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
</match>

My objective is to forward the logs to Elasticsearch and use Kibana dashboard to filter and analyze.

I have already tried using a lower version on the fluentd-elasticsearch plugin but at that time the agent did not start at all. Have tried the same conf in another non-hardened RHEL7 machine.

Upvotes: 1

Views: 1104

Answers (1)

Manish
Manish

Reputation: 21

I found the solution in a different thread:

"curl: (52) Empty reply from server" / timeout when querying ElastiscSearch

What made me search this thread was the Curl output:

    # curl -i -X --HEAD http://127.0.0.1:9200/
    curl: (52) Empty reply from server

Changed Xpack security parameter in /etc/elasticsearch/elasticsearch.yml2 from true to false:

    xpack.security.enabled: false

Restarted ES service and the errors in td-agent logs were gone.

Upvotes: 1

Related Questions