CODEWITHSUNDEEP
mongodbreplication
Sunil Jose
Sunil Jose

Reputation: 325

Mongodb : Primary was down for a day, when turned up old data not synching, current data synching

I have a mongodb replica set with one primary and two secondary, primary were down for one day, due to some os issues. Then one of the secondary elected as primary. Used that secondary as primary and used in application. After one day, old primary turned up and currently it is using as secondary. It seems only current data are updating in this secondary not synching the data of the downtime day. Is there any way to know, it is synching in background? if not synching what i have to do? I need to convert old primary again as primary, it can be done only if all data get synched.

Upvotes: 0

Views: 337

Answers (1)

R2D2
R2D2

Reputation: 10727

If your old PRIMARY become SECONDARY this mean it successfully replicated the missed data from your new PRIMARY ... , exception is made only for the writes that exist in your old PRIMARY immediately before the election not able to replicate to your new PRIMARY. Those writes you can locate in your "rollback" folder in your old PRIMARY in bson format , you will need to review those bson files and check manually if you need to re-insert or consider them again.

Troubleshooting steps to follow:

  1. Check the current replicaSet status with:

    rs.status()

  2. If you have 1x PRIMARY + 2x SECONDARY it seems everything to be fine , your members are in sync to the PRIMARY.

  3. If some of the members are in different state they maybe init syncing and you will need to wait for some time until the process finish or there is other problem so you may need to wait abit.

  4. If for some reason the members do not init sync succesfully , you can force fresh init sync so they try from the scratch.

Here are simple steps to start init sync:

4.1. Stop the member:

   mongo --p XXX
   use admin
   db.shutdownServer()

4.2. Go the data folder and remove anything inside:

   cd /my_member_data_folder/
   rm -rf *

4.3. Start the member again and wait until init sync and successfully move to SECONDARY state.

   mongod --config my_config_file.conf

  1. In case everything is fine and you just need to switch the replicaSet to your old PRIMARY you may do so by just re-configure the priority of this member as follow:

    newPRIMARY>var x=rs.conf()
newPRIMARY>x.members[0].priority=10
newPRIMARY>rs.reconfig(x)

Supposing members[0] is the old PRIMARY , and the rest of member they have the priority < 10

Upvotes: 1

Related Questions