"MongoError: no mongos proxy available" when connecting to Replica Set

Question

I'm following this tutorial (https://github.com/drginm/docker-boilerplates/tree/master/mongodb-replicaset) in order to get a mongodb replicaset of three instances to work in docker-compose.

Here are the steps I've taken so far:

1) I've copied the setup and mongo-rs0-1 folders into my root directory.

2) I've added the three mongo instances and the setup instance into my docker-compose file. It now looks like this:

version: '3'

services:
  mongo-rs0-1:
    image: "mongo-start"
    build: ./mongo-rs0-1
    ports:
      - "27017:27017"
    volumes:
      - ./mongo-rs0-1/data:/data/db
    networks:
      - app-network
    depends_on:
      - "mongo-rs0-2"
      - "mongo-rs0-3"
  mongo-rs0-2:
    image: "mongo"
    command: --replSet rs0 --smallfiles --oplogSize 128
    networks:
      - app-network
    ports:
      - "27018:27017"
    volumes:
      - ./mongo-rs0-2/data:/data/db
  mongo-rs0-3:
    image: "mongo"
    command: --replSet rs0 --smallfiles --oplogSize 128
    networks:
      - app-network
    ports:
      - "27019:27017"
    volumes:
      - ./mongo-rs0-3/data:/data/db
  setup-rs:
    image: "setup-rs"
    build: ./setup
    networks:
      - app-network
    depends_on:
      - "mongo-rs0-1"
  nodejs:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    container_name: nodejs
    restart: unless-stopped
    networks:
      - app-network
    depends_on:
      - setup-rs
  nextjs:
    build:
      context: ../.
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    container_name: nextjs
    restart: unless-stopped
    networks:
      - app-network
    depends_on:
      - nodejs
  webserver:
    image: nginx:mainline-alpine
    container_name: webserver
    restart: unless-stopped
    ports:
      - "80:80"
    volumes:
      - ./picFolder:/picFolder
      - web-root:/var/www/html
      - ./nginx-conf:/etc/nginx/conf.d
      - certbot-etc:/etc/letsencrypt
      - certbot-var:/var/lib/letsencrypt
    depends_on:
      - nodejs
      - nextjs
      - setup-rs
    networks:
      - app-network

volumes:
  certbot-etc:
  certbot-var:
  web-root:
    driver: local
    driver_opts:
      type: none
      device: /
      o: bind

networks:
  app-network:
    driver: bridge  

3) This hasn't require modification of my nginx.conf file, but I've added it here for parsimony:

server {
    listen 80;
    listen [::]:80;

    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;

    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;

    server_name example.com www.example.com localhost;

    location /socket.io {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $host;
      proxy_pass http://nodejs:8000/socket.io/;
    }

    location /back { 
      proxy_connect_timeout 75s;
      proxy_read_timeout 75s;
      proxy_send_timeout 75s;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
      proxy_pass http://nodejs:8000/back/;
    }

    location /staticBack{
      alias /picFolder;
      expires 1y;
      access_log off;
      add_header Cache-Control "public";
    }

    location / {
      proxy_pass http://nextjs:3000;
    }

    location ~ /.well-known/acme-challenge {
      allow all;
      root /var/www/html;
    }
}

4) Finally, I've changed my connection string to 'mongodb://mongo-rs0-1,mongo-rs0-2,mongo-rs0-3/test' as shown here (https://github.com/drginm/docker-boilerplates/blob/master/mongodb-replicaset/web-site/database.js)

This all seems correct, but my nodejs throws the following error when attempting to connect to Mongoose

mongoose.connect("mongodb://mongo-rs0-1,mongo-rs0-2,mongo-rs0-3/test");

MongoDB connection error: { MongoError: no mongos proxy available
    at Timeout.<anonymous> (/var/www/back/node_modules/mongodb-core/lib/topologies/mongos.js:757:28)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5) name: 'MongoError', [Symbol(mongoErrorContextSymbol)]: {} }

Does anyone see what is going on? I've tried searching for this issue, but it looks like the generic answer is that somehow mongodb just isn't 'seeing' the instances. I'm a little lost, any help would be appreciated.

EDIT:

After a little more digging I found this SO (Mongoose connection to replica set) and found that the mongo.conf file (https://github.com/drginm/docker-boilerplates/blob/master/mongodb-replicaset/mongo-rs0-1/mongo.conf) appears to say that the replica set name is rs0. So now I connect with:

mongoose.connect("mongodb://mongo-rs0-1,mongo-rs0-2,mongo-rs0-3/test?replicaSet=rs0");

However, I am still getting the following error (at least different!):

MongoDB connection error: { MongoError: no primary found in replicaset or invalid replica set name
    at /var/www/back/node_modules/mongodb-core/lib/topologies/replset.js:636:11
    at Server.<anonymous> (/var/www/back/node_modules/mongodb-core/lib/topologies/replset.js:357:9)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:116:13)
    at Server.emit (events.js:211:7)
    at /var/www/back/node_modules/mongodb-core/lib/topologies/server.js:508:16
    at /var/www/back/node_modules/mongodb-core/lib/connection/pool.js:532:18
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickCallback (internal/process/next_tick.js:181:9) name: 'MongoError', [Symbol(mongoErrorContextSymbol)]: {} }

I'm pretty sure that the replica set name is now correct, but I think maybe I have to specify which is primary? However this SO (https://dba.stackexchange.com/questions/136621/how-to-set-a-mongodb-node-to-return-as-the-primary-of-a-replication-set) seems to imply that this should happen automagically. Any help still appreciated.

EDIT EDIT:

Further research has found this SO post (Mongoose connection to replica set). Given the newer v5 mongodb the following options connect now seems to be the best bet:

var options = {
  native_parser: true,
  auto_reconnect: false,
  poolSize: 10,
  connectWithNoPrimary: true,
  sslValidate: false,
  socketOptions: {
      keepAlive: 1000,
      connectTimeoutMS: 30000
  }
};

mongoose.connect("mongodb://mongo-rs0-1:27017,mongo-rs0-2:27017,mongo-rs0-3:27017/test?replicaSet=rs0", options);

connectWithNoPrimary: true seems especially important, as if nodejs has a race condition with the mongo services booting from Docker they may not have yet elected a primary.

However, I am still getting the following error (again, subtly different):

MongoDB connection error: { MongoError: no secondary found in replicaset or invalid replica set name
    at /var/www/back/node_modules/mongodb-core/lib/topologies/replset.js:649:11
    at Server.<anonymous> (/var/www/back/node_modules/mongodb-core/lib/topologies/replset.js:357:9)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:116:13)
    at Server.emit (events.js:211:7)
    at /var/www/back/node_modules/mongodb-core/lib/topologies/server.js:508:16
    at /var/www/back/node_modules/mongodb-core/lib/connection/pool.js:532:18
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickCallback (internal/process/next_tick.js:181:9) name: 'MongoError', [Symbol(mongoErrorContextSymbol)]: {} }

So now it cannot find the secondary replica set. Adding connectwithnosecondary doesn't do anything and results in the same error - I don't think it's a valid option. Still stuck, any help would be appreciated.

EDIT EDIT EDIT:

I've changed my nodejs connect function to exist in regress callback on error. My hope was that this would simply continue to try connecting until any potential race condition in docker-compose was resolved and I could connect successfully. However, I keep getting the above error MongoError: no secondary found in replicaset or invalid replica set name. So I no longer think the problem is a race condition on connect - at least that doesn't appear to be the current error.

  var options = {
    native_parser: true,
    auto_reconnect: false,
    poolSize: 10,
    connectWithNoPrimary: true,
    sslValidate: false
  };

  // mongoose.connect("mongodb://mongo-rs0-1,mongo-rs0-2,mongo-rs0-3/?replicaSet=rs0", {  useNewUrlParser: true, connectWithNoPrimary: true });
  const connectFunc = () => {
    mongoose.connect("mongodb://mongo-rs0-1:27017,mongo-rs0-2:27017,mongo-rs0-3:27017/test?replicaSet=rs0", options);
    mongoose.Promise = global.Promise;
    var db = mongoose.connection;
    db.on('error', (error)=>{
      console.log('MongoDB connection error:', error)
      console.log('now calling connectFunc() again');
      connectFunc()
    });
    db.once('open', function() {
      // we're connected!
      console.log('connected to mongoose db')
    });

  }

  connectFunc()

EDIT x4:

Here is another example tutorial that I've gotten to the same error with: https://gist.github.com/patientplatypus/e48e6efdcc9f0f1aa551cc8342d0f2f3. I'm now half convinced that this may be a bug with automattic/mongoose so I've opened a bug report which can be viewed here: https://github.com/Automattic/mongoose/issues/7705. This may be related to this long standing issue that was filed in 2016: https://github.com/Automattic/mongoose/issues/4596.

EDIT x5:

Here is the log output for one of mongo containers:

patientplatypus:~/Documents/patientplatypus.com/forum/back:19:47:11$docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                      NAMES
d46cfb5e1927        nginx:mainline-alpine   "nginx -g 'daemon of…"   3 minutes ago       Up 3 minutes        0.0.0.0:80->80/tcp         webserver
6798fe1f6093        back_nextjs             "npm start"              3 minutes ago       Up 3 minutes        0.0.0.0:3000->3000/tcp     nextjs
ab6888f703c7        back_nodejs             "/docker-entrypoint.…"   3 minutes ago       Up 3 minutes        0.0.0.0:8000->8000/tcp     nodejs
48131a82b34e        mongo-start             "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes        0.0.0.0:27017->27017/tcp   mongo1
312772b1b0f1        mongo                   "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes        0.0.0.0:27019->27017/tcp   mongo3
9fe9a16eb20e        mongo                   "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes        0.0.0.0:27018->27017/tcp   mongo2
patientplatypus:~/Documents/patientplatypus.com/forum/back:19:48:55$docker logs 9fe9a16eb20e
2019-04-12T00:45:29.689+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-04-12T00:45:29.727+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=9fe9a16eb20e
2019-04-12T00:45:29.728+0000 I CONTROL  [initandlisten] db version v4.0.8
2019-04-12T00:45:29.728+0000 I CONTROL  [initandlisten] git version: 9b00696ed75f65e1ebc8d635593bed79b290cfbb
2019-04-12T00:45:29.728+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2019-04-12T00:45:29.728+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-04-12T00:45:29.728+0000 I CONTROL  [initandlisten] modules: none
2019-04-12T00:45:29.729+0000 I CONTROL  [initandlisten] build environment:
2019-04-12T00:45:29.729+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-04-12T00:45:29.729+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-12T00:45:29.729+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-12T00:45:29.729+0000 I CONTROL  [initandlisten] options: { net: { bindIpAll: true }, replication: { oplogSizeMB: 128, replSet: "rs" }, storage: { mmapv1: { smallFiles: true } } }
2019-04-12T00:45:29.734+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2019-04-12T00:45:29.738+0000 I STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-04-12T00:45:29.741+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-04-12T00:45:29.742+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1461M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-04-12T00:45:43.165+0000 I STORAGE  [initandlisten] WiredTiger message [1555029943:165420][1:0x7f7051ca9a40], txn-recover: Main recovery loop: starting at 7/4608 to 8/256
2019-04-12T00:45:43.214+0000 I STORAGE  [initandlisten] WiredTiger message [1555029943:214706][1:0x7f7051ca9a40], txn-recover: Recovering log 7 through 8
2019-04-12T00:45:43.787+0000 I STORAGE  [initandlisten] WiredTiger message [1555029943:787329][1:0x7f7051ca9a40], txn-recover: Recovering log 8 through 8
2019-04-12T00:45:43.849+0000 I STORAGE  [initandlisten] WiredTiger message [1555029943:849811][1:0x7f7051ca9a40], txn-recover: Set global recovery timestamp: 0
2019-04-12T00:45:43.892+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-04-12T00:45:43.972+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2019-04-12T00:45:43.972+0000 I CONTROL  [initandlisten] 
2019-04-12T00:45:43.972+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-04-12T00:45:43.972+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-04-12T00:45:43.973+0000 I CONTROL  [initandlisten] 
2019-04-12T00:45:44.035+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-04-12T00:45:44.054+0000 I REPL     [initandlisten] Did not find local voted for document at startup.
2019-04-12T00:45:44.064+0000 I REPL     [initandlisten] Rollback ID is 1
2019-04-12T00:45:44.064+0000 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2019-04-12T00:45:44.065+0000 I CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2019-04-12T00:45:44.065+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2019-04-12T00:45:44.069+0000 I CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
2019-04-12T00:45:45.080+0000 I FTDC     [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK

This line in particular

Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset

looks worrying and I'll have to keep looking into it. If anyone sees anything please let me know.

EDITx6

So I docker exec'd into one of the mongo replicas and output rs.status(). I used docker exec -it mongo1 mongo and then rs.status() and got this output:

{
    "operationTime" : Timestamp(0, 0),
    "ok" : 0,
    "errmsg" : "no replset config has been received",
    "code" : 94,
    "codeName" : "NotYetInitialized",
    "$clusterTime" : {
        "clusterTime" : Timestamp(0, 0),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

This seems very similar to the error in the mongodb replica log above (2019-04-12T00:45:44.064+0000 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset). Does anyone know what it thinks is missing?

Answer 1

I got this error when trying to connect to a replica set, using Node.js 0.10/mongodb 2.2.36. The error is caused by the proxy requiring TLS 1.2.

To solve it, upgrade your Node.js runtime to 0.12 or newer.

Answer 2

probably you are using replica set, so your connection string should be adjusted. try to add the param replicaSet=rs0 in the end of your connection string so it would be for ex.

mongodb://<usr>:<pass>@<host1>:<port1>,<host2>:<port2>/<database_name>?replicaSet=rs0<&otherParams>

Answer 3

I had a similar issue. I was constantly getting the "MongoError: no mongos proxy available" error. Finally I managed to solve it this way:

// The mongoose connection options
let options = {
  ssl: true,
  sslValidate: true,
  poolSize: 1,
  reconnectTries: 1,
  useNewUrlParser: true,
  dbName: 'myDb' // Specify the database here
}

And my connection string looks something like this:

mongodb://<usr>:<pass>@host1.com:12345,host2.com:12346/adminDb?authSource=admin&ssl=true

I left the default admin DB in the connection string and instead, defined the DB I wanted to connect to in the options. That solved the issue.

Answer 4

You need to initialise the replica set in order to be able to access it. Without that, applications won't be able to connect to your local instances.

Ideally, you need to add some time between the replica set configuration (the setup-rs step) and the next one, as the replica set configuration can take longer than the application startup time.

Or fix the script, if the script itself is faulty.