CODEWITHSUNDEEP
azureelasticsearchterraformterraform-provider-azure
yanis
yanis

Reputation: 333

The right approach to do Elasticsearch upgrades via terraform

I would like to discuss what are the best practices/approaches engineers do while upgrading elasticsearch clusters. I believe this post may serve as a good example of strategies and steps to perform, guaranteeing no data loss, minimum downtime, scalability and availability of the elasticsearch services.

To start the initiative, we can break the upgrade into two subsections:

1) Performing upgrade on master nodes:

Since master nodes do not contain any data and are responsible for controlling the cluster I believe we can safely do terraform apply to add all the upgraded master node VMs and then remove the old ones.

2) Performing upgrade on data nodes:

As many people already know, there is certain limitation on the ability to update data nodes. We cannot afford to completely deallocate the VM and replace it with another. A good practice in my opinion is to:

a) Stop the index allocation to the old VM

b) Then performing terraform apply to create the new upgraded version of the data node VM(and manually modifying the terraform state in order the old VM not to be destroyed)

c) Allowing traffic(index creation) to the new VM and using the elasticsearch APIs to transfer the data from the old to the new VM

d) Manually changing the terraform state allowing it to delete the old VM.

These are just idealistic steps, I would like to see your opinion and strategies to perform safe elasticsearch upgrades via Terraform.

Upvotes: 1

Views: 697

Answers (1)

Dave Turner
Dave Turner

Reputation: 1896

The reference manual has guidelines regarding removing master-eligible nodes that you must respect in versions 7 and later. It's much trickier to get this right in earlier versions because of the discovery.zen.minimum_master_nodes setting.

Your strategy for data nodes sounds slow and expensive given that you might be moving many terabytes of data around for each node. It's normally better to restart/upgrade larger data nodes "in place", detaching and reattaching the underlying storage if needed.

Upvotes: 1

Related Questions