CODEWITHSUNDEEP
mongodbmongodb-replica-set
fguillen
fguillen

Reputation: 38888

MongoDB: replica set and round robin query distribution?

We have a MongoDB Replica Set with this structure:

We have an application that execute very heavy queries using the readonly secondaries.

The problem is that the system is overloading one of the instances and the other 2 are mostly lazy, they still receive queries but it is clear that the system has a preferred instance and therefore the structure is not scaling properly.

This is our connection configuration:

mongo:
  collection: "panelUsers"
  urls:
    - "mongo-001.my.network:27017"
    - "mongo-002.my.network:27017"
    - "mongo-readonly-001.my.network:27017"
    - "mongo-readonly-002.my.network:27017"
    - "mongo-readonly-003.my.network:27017"
  connection_opts:
    user: "user"
    password: "password"
    database: "mydb"
    replica_set: "mongo-replica"
    read:
      mode: "secondary"
      tag_sets:
        - nodeType: "readonly"

I can see in the monitors how one of the instances are taking over the big portion of the load:

enter image description here

When the process starts, the 3 instances look like they are taking a fair portion of the queries, but as time progresses one of the instances starts taking the most of it. When the 3 instances are collaborating properly I can see how the system performance (queries per minute) is much better, when one of the instances is taking most of the load the whole system starts slowing down.

For tests purposes I shutdown the instances taking the most of the load and I see the other 2 start taking a fair amount of the load, then the system is faster that when the 3 instances are running but only one is taking most of the load. After sometime 1 instance (from the 2 left) starts taking all of the load and the process starts slowing down.

I have read about the Server Selection Algorithm and I understood that Mongo is selecting the instance to solve the query based on ping latency. Doesn't look like a very evolved system :/

I think in my case a Round Robin mechanism would be better.

Can I activate a Round Robin server selection algorithm in our MongoDB Read Replicas?

PS: I have read that sharding is a better solution for my case but it doesn't look like something I can do easily, before I go for this I would like to know if there is an easier solution for my case.

Upvotes: 0

Views: 1014

Answers (1)

Yahya
Yahya

Reputation: 3444

First of all, MongoDB only has 1 primary at a time. It works as a single master.

All read/writes come through primary, unless you specify read preference from a secondary.

Your "read-only" secondaries. Have you configured them to be hidden? Because if failover happens, they could also become a primary.

What I'd suggest is to do the following:

  1. Instead of 5, keep a 3 nodes replica set (for operational load). With standard indexes for your "usual" reads and writes
  2. Have 2 dedicated secondaries (hidden nodes) for specialist load, where you can create indexes to support your queries
  3. Make sure you are not doing collection scan or in memory sort. See https://docs.mongodb.com/manual/reference/method/cursor.explain/
  4. Having additional nodes in replica set, doesn't help you with higher read/write throughput because whatever is being written on Primary, is also being written on Secondary. So it is as busy as Primary too. But you can create specialist hidden nodes to support reporting load as explained in 2.
  5. Sharding is the right way to increase read/write throughput

Upvotes: 1

Related Questions