CODEWITHSUNDEEP
elasticsearchlogstashelk
RAM NATHAN
RAM NATHAN

Reputation: 19

Logstash shuts down during clustering

Im having three 7.4.2 ELK nodes.I have configured clustering in first elasticsearch/logstash/kibana node and restarted ELK in that node. It wass successful. But once I have configured clustering in second ELK node, first node logstash stopped automatically with below error

An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::
    HttpClient::Pool::HostUnreachableError: Could not reach host Manticore::SocketException:
    Connection refused (Connection refused)>, :backtrace=>["/opt///logstash/vendor
    /bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/
    elasticsearch/http_client/pool.rb:293:in perform_request_to_url'", "/opt/****/****/logstash /vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs /elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/opt///
    logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/
    logstash/outputs/elasticsearch/http_client/pool.rb:373:in with_connection'", " /opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib /logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'",
    "/opt///logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-
    java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in block in Pool'", "/opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0- java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in get'", "/opt///logstash/
    ndor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/
    outputs/elasticsearch/http_client.rb:378:in get_xpack_info'", "/opt/****/****/logstash/ vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/ outputs/elasticsearch/ilm.rb:57:in ilm_ready?'", "/opt///logstash/vendor/bundle
    /jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/
    ilm.rb:28:in ilm_in_use?'", "/opt/****/****/logstash/vendor/bundle/jruby/2.5.0/gems/ logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in block in setup_after_successful_connection'"]}

There is a FATAL error in logs as below [2020-10-12T17:52:25,998][ERROR][logstash.outputs.elasticsearch][events] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://...:9200/][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=>["logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:293:in perform_request_to_url'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in with_connection'", logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in block in Pool'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:162:in get'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:378:in get_xpack_info'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:57:in ilm_ready?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:28:in ilm_in_use?'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:14:in install_template'", logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:130:in install_template'", "logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:51:in block in setup_after_successful_connection'"]} Any idea? Does it have something to do with ilm_enabled? I have made clustering changes in elasticsearch as below.

cluster.name: "elasticsearch"
    node.name: "node-2"
    node.data: true
    path.logs: "/var/opt/logs/elasticsearch/"
    path.data: "/var/lib/elasticsearch"
    network.host: "**.**.**.**"
    http.port: 9200
    discovery.seed_hosts:
    - "**.**.**.**"
    - "**.**.**.**"
    cluster.initial_master_nodes:
    - "node-1"
    - "node-2"

And Im using elasticsearch output plugin , where I added clustering nodes as below.

elasticsearch 
            { 
       hosts => ["**.**.**.**:9200","**.**.**.**:9200"]
               document_id => "%{authsid}"
               index =>  "dashboard_write"
               script => "ctx._source.loginCount= params.event.get('loginCount');
                       ctx._source.contractName= params.event.get('contractName');
                        ctx._source.userName= params.event.get('userName');
                         ctx._source.sessionID= params.event.get('sessionID');
                         ctx._source.eventID= params.event.get('eventID');"
              doc_as_upsert => "true"
               action => "update"  
              ilm_enabled => false             
        }

I have done above changes in first node elasticsearch.yml & logstash output.conf files and restarted first node. It came up successfully. Then I have done same changes in second node and restarted second node, then first node logstash went down automatically

Upvotes: 0

Views: 592

Answers (1)

Orkun
Orkun

Reputation: 512

In my point of view, your logstash configuration doesn't find the index template. You have ilm_settings in elasticsearch output plugin but you didn't specify the index template. So, logstash trying to find wrong end point. Thats why you get an error. Use similar like below,

elasticsearch {
    
    hosts => ["localhost:9200"]
    index => "index_name"
    manage_template => true
    template_overwrite => true
    ilm_enabled => false
    template_name => "template_name"
    template => "path_to_template"
    document_id => "document_id" 
    http_compression => true
}

Upvotes: 1

Related Questions