CODEWITHSUNDEEP
cluster-computingmarklogicfailoverreindex
Manish Joisar
Manish Joisar

Reputation: 1310

MarkLogic Re-indexing in progress & Forest replicas changeover

MarkLogic 9.0.8.2

We have configured MarkLogic cluster (3 nodes) as follows PF - primary forest, RF - replica forest

Host1: PF01 RF02 RF03 Host2: PF02 RF01 RF03 Host3: PF03 RF01 RF02

In normal scenario, when we see database status page, PF should be in "open" state and and everything else will be "wait replicating" state.

Now due to some reason, state of RF01 and RF02 became in "open" state, so that means all load is now on Host3

Reindexing is in progress.

Now question is while reindexing is in progress, can we restart RF01-2 & RF02-2, so that PF01 and PF02 are again in "open" state?

Are there any best practices we should follow for forest restart?

Upvotes: 1

Views: 123

Answers (2)

Mads Hansen
Mads Hansen

Reputation: 66783

See this knowledgbase article What Triggers Failover in MarkLogic Server

If this occurred during a reindex, I suspect that you may not have sufficient resources to support the workload. Depending on what "normal" is for your system, reindexing can cause an increase in resource demands.

Look through your ErrorLog.txt and I suspect you will see signs of trouble. Look for any log events with a severity greater than INFO (NOTICE, WARNING, ERROR, CRITICAL).

You may need to reduce the throttle of your reindexing and consider scaling options to ensure that your cluster is appropriately sized. You might consider opening a MarkLogic Support ticket, if you are eligible.

If you think that the primary hosts are ready to take over, you can "flip the forests" back to the primary by restarting the replica forests that are now open.

See the documentation: Reverting a Failed Over Forest Back to the Primary Host

Also, the following knowledgebase article provides a script that can be executed in order to flip all of the open replicas:

https://help.marklogic.com/Knowledgebase/Article/View/427/0/scripting-failover-flipping-replica-forests-back-to-their-masters-using-xquery

Upvotes: 3

bosari
bosari

Reputation: 2000

I believe, you should wait until your re-indexing is complete. Once the re-indexing is completed you can flip back forests. As re-indexing started when master was in open state its possible that some documents might not be synced in master forests and re-indexing completed or failed. Those documents might get missed.

Upvotes: 2

Related Questions