Elasticsearch Cluster On EC2 Aws Fails to join cluster

Question

I installed oracle-jdk8 and elasticsearch on a ec2 instance and created an ami out of it. In the next copy of the ec2 machine i just changed the node name in elasticsearch.yml However both the nodes if run individually are running.[NOTE the node id is appearing as same] But if run simultaneously, the one started later is failing with following in the logs:

[2018-08-07T16:35:06,260][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{uQHBhDuxTeWOgmZHsuaZmA}{akWOcJ47SZKpR_EpA2lpyg}{10.127.114.212}{10.127.114.212:9300}{aws_availability_zone=us-east-1c, ml.machine_memory=66718932992, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[node-2][10.127.114.212:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {node-1}{uQHBhDuxTeWOgmZHsuaZmA}{Ba1r1GoMSZOMeIWVKtPD2Q}{10.127.114.194}{10.127.114.194:9300}{aws_availability_zone=us-east-1c, ml.machine_memory=66716696576, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, found existing node {node-2}{uQHBhDuxTeWOgmZHsuaZmA}{akWOcJ47SZKpR_EpA2lpyg}{10.127.114.212}{10.127.114.212:9300}{aws_availability_zone=us-east-1c, ml.machine_memory=66718932992, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} with the same id but is a different node instance];

My elasticsearch.yml:

cluster.name: elk
node.name: node-1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true

network.host: 0.0.0.0
network.publish_host: _ec2:privateIp_
transport.publish_host: _ec2:privateIp_

discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ElasticSearch: elk-tag

cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

Output from _nodes endpoint:

//----Output when node-1 is run individually/at first----
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elk",
  "nodes" : {
    "uQHBhDuxTeWOgmZHsuaZmA" : {
      "name" : "node-1",
      "transport_address" : "10.127.114.194:9300",
      "host" : "10.127.114.194",
      "ip" : "10.127.114.194",
      "version" : "6.3.2",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "053779d",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "aws_availability_zone" : "us-east-1c",
        "ml.machine_memory" : "66716696576",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 3110,
        "mlockall" : true
      }
    }
  }
}
//----Output when node-2 is run individually/at first----
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elk",
  "nodes" : {
    "uQHBhDuxTeWOgmZHsuaZmA" : {
      "name" : "node-2",
      "transport_address" : "10.127.114.212:9300",
      "host" : "10.127.114.212",
      "ip" : "10.127.114.212",
      "version" : "6.3.2",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "053779d",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "aws_availability_zone" : "us-east-1c",
        "ml.machine_memory" : "66718932992",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 4869,
        "mlockall" : true
      }
    }
  }
}

Answer 1

Solved this by deleting rm -rf /var/lib/elasticsearch/nodes/ in every instance and restarting elasticsearch.