CODEWITHSUNDEEP
mongodb
Shail Patel
Shail Patel

Reputation: 1814

mongodb replica set unreachable

I am trying to configure a standalone mongodb replica set with 3 instances. I seem to have gotten into a funky state. Two of my instances went down, and I was left with all secondary nodes. I tried to follow this: http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

I got this error though: 

 rs0:SECONDARY> rs.reconfig(cfg, {force : true})
 {
    "errmsg" : "exception: need most members up to reconfigure, not ok : obfuscated_hostname:27019",
    "code" : 13144,
    "ok" : 0
 }

When I look at the logs I see this: 

 Fri Aug  2 20:45:11.895 [initandlisten] options: { config: "/etc/mongodb1.conf",
 dbpath: "/var/lib/mongodb1", logappend: "true", logpath: "/var/log/mongodb/mongodb1.log",
 port: 27018, replSet: "rs0" }
 Fri Aug  2 20:45:11.897 [initandlisten] journal dir=/var/lib/mongodb1/journal
 Fri Aug  2 20:45:11.897 [initandlisten] recover begin
 Fri Aug  2 20:45:11.897 [initandlisten] recover lsn: 0
 Fri Aug  2 20:45:11.897 [initandlisten] recover /var/lib/mongodb1/journal/j._0
 Fri Aug  2 20:45:11.899 [initandlisten] recover cleaning up
 Fri Aug  2 20:45:11.899 [initandlisten] removeJournalFiles
 Fri Aug  2 20:45:11.899 [initandlisten] recover done
 Fri Aug  2 20:45:11.923 [initandlisten] waiting for connections on port 27018
 Fri Aug  2 20:45:11.925 [websvr] admin web console waiting for connections on port     28018
 Fri Aug  2 20:45:11.927 [rsStart] replSet I am hostname_obfuscated:27018
 Fri Aug  2 20:45:11.927 [rsStart] replSet STARTUP2
 Fri Aug  2 20:45:11.929 [rsHealthPoll] replset info hostname_obf:27017 thinks that we  are down
 Fri Aug  2 20:45:11.929 [rsHealthPoll] replSet member hostname_obf:27017 is up
 Fri Aug  2 20:45:11.929 [rsHealthPoll] replSet member hostname_obf:27017 is now in state SECONDARY
 Fri Aug  2 20:45:12.587 [initandlisten] connection accepted from ip_obf:52446 #1 (1 connection now open)
 Fri Aug  2 20:45:12.587 [initandlisten] connection accepted from ip_obf:52447 #2 (2 connections now open)
 Fri Aug  2 20:45:12.588 [conn1] end connection ip_obf:52446 (1 connection now open)
 Fri Aug  2 20:45:12.928 [rsSync] replSet SECONDARY

I'm unable to connect to the mongo instances, even though the logs say that it is up and running. Any ideas on what to do here?

Upvotes: 2

Views: 7909

Answers (2)

Jerry
Jerry

Reputation: 8111

Yes you can turn up a single secondary replica to primary if the secondary server is running fine.Do follow the below simple steps:

Step 1: Connect to member and check the current configuration

rs.conf()

Step 2: Save the current configuration to another variable.

x = rs.conf()

Step 3: Select the id,host and port of the member that is to be made as primary.

x.members = [{"_id":1,"host" : "localhost.localdomain:27017"}]

Step 4: Reconfigure the new replica set by force.

rs.reconfig(x, {force:true})

Now the desired member will be promoted as the primary.

Upvotes: 0

drmirror
drmirror

Reputation: 3760

You did not mention which version of mongodb you are using, but I assume it is post-2.0.

I think the problem with your forced reconfiguration is that after this reconfiguration, you still need to have the minimum number of nodes for a functioning replica set, i.e. 3. But since you originally had 3 members and lost 2, there is no way you could turn that single surviving node into a functioning replica set.

Your only option for recovery would be to bring up the surviving node as a stand-alone server, backup the database, and then create a new 3-node replica set with that data.

Upvotes: 3

Related Questions