CODEWITHSUNDEEP
elasticsearch
Hooman Bahreini
Hooman Bahreini

Reputation: 15559

Configuring Elasticsearch 7 cluster

I have three server and I have installed elasticsearch on all of them.

In the elasticsearch.yml I have the following config:

first server: 172.31.1.1

# /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-1
network.host: 172.31.1.1
http.port: 9200
network.host:
discovery.seed_hosts: ["172.31.1.1", "172.31.1.2", "172.31.1.3"]
cluster.initial_master_nodes:["es-1", "es-2", "es-3"]

gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# some more default values

second server: 172.31.1.2

node.name: es-2
network.host: 172.31.1.2
# everything else same as first server

second server: 172.31.1.3

node.name: es-3
network.host: 172.31.1.3
# everything else same as first server

Now when I run: curl http://es-1:9200/_cluster/stats

I notice that the cluster size is 1. if I run the same curl command on a different server, again the cluster size is 1 and the cluster_name is the same, but cluster_uuid is different...

It seems that nodes are not able to connect to each other...

I am using Ubuntu server, for node name I edited /etc/hostname and changed the host name to es-1 and used the same name in elasticsearch.yml... but it seems that cluster nodes are not able to communicate with one another...

Update 1

curl http://es-1:9200/_cat/health
prod-es-cluster red 1 1 0 0 0 0 0 0 - NaN%

curl http://es-1/:9200/_cat/nodes
172.31.1.1 8 38 1 0.03 0.03 0.00 mdi * ip-172-31-1-1

I just check the log and it's strange... 

sudo cat /var/log/elasticsearch/elasticsearch.log
[INFO ][o.e.n.Node               ] [es-1] stopping ...
[INFO ][o.e.x.w.WatcherService   ] [es-1] stopping watch service, reason [shutdown initiated]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es-1] [controller/19159] [Main.cc@148] Ml controller exiting
[INFO ][o.e.x.m.p.NativeController] [es-1] Native controller process has stopped - no new native processes can be started
[INFO ][o.e.n.Node               ] [es-1] stopped
[INFO ][o.e.n.Node               ] [es-1] closing ...
[INFO ][o.e.n.Node               ] [es-1] closed

That is all written to the log file... after restarting Elasticsearch nothing new gets added to the log file... it feels like I am either reading the wrong log file or I have put the configuration in the wrong .yml file

Update 2

This is how I start the service:

sudo /bin/systemctl enable elasticsearch.service

Update 3

As pointed out by @hamidbayat, I think there is permission issues. Only super user have permission to /var/lib/elasticsearch

If I try to run elastic search as my ubuntu user I will get this error:

ubuntu@my-ip:~$ /usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch-env: line 73: 
/etc/default/elasticsearch: Permission denied

If I try to run it as root, I will get this error:

ubuntu@myip:~$ sudo /usr/share/elasticsearch/bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[data][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
        ... 6 more

Upvotes: 2

Views: 2369

Answers (2)

Hooman Bahreini
Hooman Bahreini

Reputation: 15559

This is how I solved the issue:

  1. I was reading elasticsearch.log and thought I am not getting anything in the log file, but realized that the logs are being written to cluster-name.log (es-cluster.log in my case)

  2. I commented out http.port: 9200 in elasticsearch.yml I believe elasticsearch uses 9300 by default.

  3. I removed the node's own IP from discovery.seed_hosts:, so only other nodes are added to this list, e.g. for first server I have:

    discovery.seed_hosts: ["172.31.1.2", "172.31.1.3"]

  4. I realized each node has formed it's own cluster and that's why they were not joining the same cluster. So I had to merge the clusters. Since I had no data in the cluster, I had the option of deleting the data path. So I left node 1 running and stopped elasticsearch on node 2 & 3. Then deleted the data path on nodes 2 & 3 (in my case data path was /var/lib/elasticsearch/). Then started nodes 2 and 3 and they joined the cluster.

Upvotes: 3

PRADEEP Kumar
PRADEEP Kumar

Reputation: 243

You need to configure bootstrap check, in 6.x it throws warning

For 7.x version bootstrap configuration is must

bootstrap.system_call_filter: false add this try starting

Upvotes: 0

Related Questions