MongoDB Replica Set Healtcheck - HostUnreachable

Question

I've been having trouble setting up my Mongo v3.2 replica set / cluster. I've included mongods logs with inline comments from A) the node I initialized the replica set from and B) another node (rest of peers are having same issue). In short, the issue is that the heartbeat from all the other nodes to the node I initially setup the replica set from keep failing due to HostUnreachable, but I'm not sure how to debug further, as mongo <initial node> works fine from all the other nodes so I don't think it's a network access problem. After some Googling, I assumed this was due to the bindIp setting I had in my Mongo config file (or the lack thereof), but listing all node IPs there doesn't seem to do the trick.

See the mongodb config files & logs here-

https://gist.github.com/tejasmanohar/f0e705fb0d9e96f68e05e1ab20c478be

Why could this healthcheck be failing and/or how I can debug further or maybe reproduce the failed connection apart outside of mongod (I can ping and connect via mongo to all peers from each host)? Thanks!

Update: options: { config: "/etc/mongod.conf", net: { bindIp: true }, replication: { replSet: "rs0" } } I noticed that the mongod logs show bindIp: true rather than a list set in my Mongo configuration file. Is that problematic? I tried providing a comma-separated string instead of an array since the type of the field is a bit unclear in docs but same result.

Answer 1

It seems you are setting the ips of the other machines on bindIp, but you should use the machine's ip that is reachable by the others.

For example, on your mongo-initial.q machine, your bindIp should be like:

"bindIp": "127.0.0.1,10.0.11.2"

More info here.

Don't know if it will fix everything but it's a start.

Answer 2

So, the solution doesn't make 100% sense to me yet, but I had to change bindIps setting to a comma-separated string instead of an array. Array seems to work in YAML but not JSON. Still don't understand why it logged out as a bool before. @andresk pointed out something else wrong about my solution-- my bindIp setting had my peer IPs but not the own host.