Can't convert standalone mongoDB to Replica Set: Connection timed out

Question

Mongo 3.2.8

I'm trying to add replica(M2) to my standalone MongoDB(M1).

There are following scripts I'm using for it:

M2: /usr/bin/mongod --dbpath /var/lib/mongodb --quiet --replSet "rs0"

M1: /usr/bin/mongod --dbpath /var/lib/mongodb --quiet --replSet "rs0"

mongo

rs.initiate()

rs.status()

{
    "set" : "rs0",
    "date" : ISODate("2016-09-22T18:13:04.380Z"),
    "myState" : 1,
    "term" : NumberLong(8),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [
        {
            "_id" : 0,
            "name" : "67.221.191.2:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 992,
            "optime" : {
                "ts" : Timestamp(1474567982, 34308),
                "t" : NumberLong(8)
            },
            "optimeDate" : ISODate("2016-09-22T18:13:02Z"),
            "electionTime" : Timestamp(1474566993, 1),
            "electionDate" : ISODate("2016-09-22T17:56:33Z"),
            "configVersion" : 294102,
            "self" : true
        }
    ],
    "ok" : 1
}

rs.conf()

{
    "_id" : "rs0",
    "version" : 294102,
    "protocolVersion" : NumberLong(1),
    "members" : [
        {
            "_id" : 0,
            "host" : "67.221.191.2:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 2,
            "tags" : {

            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatIntervalMillis" : 2000,
        "heartbeatTimeoutSecs" : 10,
        "electionTimeoutMillis" : 10000,
        "getLastErrorModes" : {

        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        },
        "replicaSetId" : ObjectId("57e3e4380a1f48aa77b6a6f0")
    }
}

After adding M2 to replica set

mongo

rs.add("209.205.218.42:27017")

I have following issue for M2 server:

2016-09-22T13:36:43.750-0400 I REPL     [ReplicationExecutor] Error in heartbeat request to 67.221.191.2:27017; HostUnreachable: Connection timed out
2016-09-22T13:36:43.750-0400 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 67.221.191.2:27017
2016-09-22T13:36:50.757-0400 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to 67.221.191.2:27017 - HostUnreachable: Connection timed out

Could you suggest why it happens?

---

SOME ADDITIONAL INFORMATION:

In M1 there is nothing interesting in logs. Just information that it can't be primary without voting, so it became secondary

After adding replica I have following result on M1:

rs.status()

{
    "set" : "rs0",
    "date" : ISODate("2016-09-22T18:53:08.501Z"),
    "myState" : 2,
    "term" : NumberLong(8),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [
        {
            "_id" : 0,
            "name" : "67.221.191.2:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 3396,
            "optime" : {
                "ts" : Timestamp(1474570214, 27),
                "t" : NumberLong(8)
            },
            "optimeDate" : ISODate("2016-09-22T18:50:14Z"),
            "infoMessage" : "could not find member to sync from",
            "configVersion" : 294103,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "209.205.219.42:27017",
            "health" : 1,
            "state" : 0,
            "stateStr" : "STARTUP",
            "uptime" : 172,
            "optime" : {
                "ts" : Timestamp(0, 0),
                "t" : NumberLong(-1)
            },
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2016-09-22T18:53:06.447Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : NumberLong(0),
            "configVersion" : -2
        }
    ],
    "ok" : 1
}

I'm not good in solving network issues any suggestions appreciated.

1) M1@ubuntu:~# tcpdump dst host 67.221.191.2

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:50:40.242522 IP ubuntu.27017 > 67.221.191.2.53638: Flags [P.], seq 1253499623:1253499895, ack 528146942, win 59, options [nop,nop,TS val 9778205 ecr 1369701227], length 272
14:50:40.242629 IP ubuntu.27017 > 67.221.191.2.53638: Flags [P.], seq 272:544, ack 1, win 59, options [nop,nop,TS val 9778205 ecr 1369701227], length 272

2) I have in my tmp folder following file:

mongodb-27017.sock

Deleting of this file doesn't change anything

Fill free to ask any additional information

Answer 1

It was network issue caused by firewall. To list out all of the active iptables rules by specification, run the iptables command with the -S option

sudo iptables -S