CODEWITHSUNDEEP
elasticsearchkubernetesgoogle-kubernetes-engineelasticsearch-5
Moritz Schmitz v. Hülst
Moritz Schmitz v. Hülst

Reputation: 3652

What is the correct setting to replicate all indices across all nodes in Elasticsearch 5.6?

Cannot upgrade at the moment and I am stuck with Elasticsearch 5.6.

Trying to get cluster-mode working on GKE with success, but one major problem. Whenever I do a rolling restart (StatefulSet) of the Elasticsearch cluster I can get the cluster into a red state from which it cannot recover. Doesn't happen all the time though and works flawlessly if I only kill one node (out of three).

For some reason the restarted node is not able to recover their primary/replica shards on startup. Or the next node is terminated to early before shard allocation has been finished?

In any way I was looking for an option where I just keep every index fully on every node (mix of primary and replica shards). Not sure how to configure this properly.

How do I configure a Elasticsearch cluster with three nodes and two indices to survive an outage of two nodes?

EDIT: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/delayed-allocation.html is probably what I need, isn't it?

Upvotes: 0

Views: 140

Answers (1)

Alez
Alez

Reputation: 2659

A StatefulSet is a way to create stateful components within a distributed systems. Thus, Pods are intended execute independently with its own state. In fact a StatefulSet have a unique ordinal index, a stable network identity, and if stops it's data are not visible from the running nodes. Here the block diagram to access the PV/PVC:

 PV - PVC - POD1 \
 PV - PVC - POD2 - StatefulSet  
 PV - PVC - POD3 /

What you cannot reach with StatefulSet (differently than Deployment) is to access the POD's data if that POD stops. This means that if you have 3 PODs: if 1 POD stops your application may continue to work, if 2 PODs stops your application likely doesn't work.

If you need to continue to access the data if some PODs stop, you probably need a Deployment, instead, where every pod access the same data on PV-PVC:

          / POD1 \
 PV - PVC - POD2 - Deployment
          \ POD3 /

So, in order to meet your requirements, try to use Deployments instead of StatefulSets.

Then you can distribute PODs on each node, following the official docs, in order to enable application availability in case 2 out of 3 nodes will not be reachable. To do that you can use the following yaml example:

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
template:
  spec:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: foo
        matchLabelKeys:
          - pod-template-hash

Upvotes: 1

Related Questions