CODEWITHSUNDEEP
amazon-web-serviceselasticsearch
Jared
Jared

Reputation: 3016

How to assign shards in AWS Elasticsearch?

enter image description here

But because I'm hosting on AWS I'm not allowed to call /_settings or /_cluster so how can I assign those shards?

The error I'm getting when I POST to /_cluster:

{
  "Message": "Your request: '/_cluster' is not allowed."
}

Upvotes: 3

Views: 4981

Answers (2)

Rotem jackoby
Rotem jackoby

Reputation: 22068

You haven't provide additional information about your system and requirements but I'll assume that:

1 ) You don't want to change the number of shards (not recommended).

2 ) You want that your cluster status to be green.

So it seems that you need to allocate more nodes to your domain.

Consider increasing the number of master nodes to 3 and make sure you split them to 3 availability zones as can be seen in the UI console below:

enter image description here

Check the allocation with:

$ curl -XGET ES_Endpoint/_cat/allocation?v

And you should see that your 8 primary shards are being allocated evenly between the master nodes:

shards   disk.indices disk.used disk.avail   disk.total     disk.percent host        ip      node
     3       16.8mb    69.2mb      9.7gb      9.8gb            0        x.x.x.x   x.x.x.x   98da03fa1
     3       11.3mb    63.7mb      9.7gb      9.8gb            0        x.x.x.x   x.x.x.x   aea0f9c9
     2       18.2mb    70.6mb      9.7gb      9.8gb            0        x.x.x.x   x.x.x.x   af7fe965

I would also recommend reading this article from AWS:

We recommend that you add three dedicated master nodes to each production Amazon ES domain. Never choose an even number of dedicated master nodes.

1 ) One dedicated master node means that you have no backup in the event of a failure.

2 ) Two dedicated master nodes means that your cluster does not have the necessary quorum of nodes to elect a new master node in the event of a failure.

A quorum is the number of dedicated master nodes / 2 + 1 (rounded down to the nearest whole number), which Amazon ES sets to discovery.zen.minimum_master_nodes when you create your domain.

In this case, 2 / 2 + 1 = 2. Because one dedicated master node has failed and only one backup exists, the cluster doesn't have a quorum and can't elect a new master.

3 ) Three dedicated master nodes, the recommended number, provides two backup nodes in the event of a master node failure and the necessary quorum (2) to elect a new master.

4 ) Four dedicated master nodes are no better than three and can cause issues if you use multiple Availability Zones.

If one master node fails, you have the quorum (3) to elect a new master. If two nodes fail, you lose that quorum, just as you do with three dedicated master nodes.

In a three Availability Zone configuration, two AZs have one dedicated master node, and one AZ has two. If that AZ experiences a disruption, the remaining two AZs don't have the necessary quorum (3) to elect a new master.

5 ) Having five dedicated master nodes works as well as three and allows you to lose two nodes while maintaining a quorum. But because only one dedicated master node is active at any given time, this configuration means paying for four idle nodes. Many users find this level of failover protection excessive.

Upvotes: 0

Val
Val

Reputation: 217254

By default, one replica shard is created per primary shard and since you have only one node, the replicas cannot be assigned.

You simply need to pass number_of_replicas:0 when you create your index

PUT /my-index
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1, 
            "number_of_replicas" : 0
        }
    }
}

If you later want to increase the number of replica shards because you add new nodes, you can do it like this:

PUT /my-index/_settings
{
    "index" : {
        "number_of_replicas" : 1
    }
}

Upvotes: 4

Related Questions