Reputation: 333
I would like to discuss what are the best practices/approaches engineers do while upgrading elasticsearch
clusters. I believe this post may serve as a good example of strategies and steps to perform, guaranteeing no data loss, minimum downtime, scalability and availability of the elasticsearch
services.
To start the initiative, we can break the upgrade into two subsections:
1) Performing upgrade on master nodes
:
Since master nodes do not contain any data and are responsible for controlling the cluster I believe we can safely do terraform apply to add all the upgraded master node VMs and then remove the old ones.
2) Performing upgrade on data nodes
:
As many people already know, there is certain limitation on the ability to update data nodes. We cannot afford to completely deallocate the VM and replace it with another. A good practice in my opinion is to:
a) Stop the index allocation to the old VM
b) Then performing terraform apply
to create the new upgraded version of the data node VM(and manually modifying the terraform state
in order the old VM not to be destroyed)
c) Allowing traffic(index creation) to the new VM and using the elasticsearch
APIs to transfer the data from the old to the new VM
d) Manually changing the terraform state
allowing it to delete the old VM.
These are just idealistic steps, I would like to see your opinion and strategies to perform safe elasticsearch upgrades via Terraform.
Upvotes: 1
Views: 697
Reputation: 1896
The reference manual has guidelines regarding removing master-eligible nodes that you must respect in versions 7 and later. It's much trickier to get this right in earlier versions because of the discovery.zen.minimum_master_nodes
setting.
Your strategy for data nodes sounds slow and expensive given that you might be moving many terabytes of data around for each node. It's normally better to restart/upgrade larger data nodes "in place", detaching and reattaching the underlying storage if needed.
Upvotes: 1