CODEWITHSUNDEEP
elasticsearchsnapshotrecovery
Charlie Echo
Charlie Echo

Reputation: 67

How can I validate a botched ElasticSearch recovery?

On an older production node I'm running ElasticSearch 6.8.0. I needed to migrate indexes from an even older node to as part of our outline to get up-to-date. These nodes are segregated - not replicating. I had been doing snapshops and recoveries in small batches for ease, but nearing the end of the project I think I bit off more than the node could chew. During the restoration of a large multi-index snapshot (500GB!) the node had a memory constraint issue and went kaput. I had feared the worst, but in order to recover I double the RAM and brought the VM back online. To my surprise, the recovery appears to have completed - all indices and shards are showing 100%! The stats of all indexes match up on both the origin node and the node being migrated to, which seems promising but my experience in our field prevents me from getting any warm and fuzzy feelings yet.

My question: Is this expected from ES - a miracle recovery by some standards? Any foolproof way of validating this? Should I be comfortable with the status and carry on, or should I close the indices that were part of the "failed" snapshot recovery and run the recovery again?

Obviously I'm not a ElasticSearch guru - this tech got dropped in my lap so I'm learning as I go.

Thanks all!

Upvotes: 1

Views: 120

Answers (1)

Charlie Echo
Charlie Echo

Reputation: 67

Normally I would delete this question, but in this case I feel answering my own question may be helpful to someone else - I truly hope it is as deleting questions isn't exactly helping the community. Hopefully there is further input to enrich this thread for future searchers.

After digging in a bit more I found that 2 of the indices were in RED status.

GET /_cat/indices\?v

After a few attempts I was still unable to recover them. Even though the rest of the indices were ok, I would rather not take any chances so I deleted all indices from the failed snapshot off the new node from Kibana using the Dev console.

DELETE Index_1,Index_2,Index_3

A new recovery of the same snapshot is currently running. I assume now that the server is appropriately spec'd for this I should not have any further issues.

POST /_snapshot/shared_repo/mig-oct30-21/_restore { "indices": "*", "include_global_state": false, "ignore_unavailable": true }

Upvotes: 1

Related Questions