Meteor MongoDB subscription delivering data in 10 second intervals instead of live

Question

I believe this is more of a MongoDB question than a Meteor question, so don't get scared if you know a lot about mongo but nothing about meteor.

Running Meteor in development mode, but connecting it to an external Mongo instance instead of using Meteor's bundled one, results in the same problem. This leads me to believe this is a Mongo problem, not a Meteor problem.

The actual problem

I have a meteor project which continuosly gets data added to the database, and displays them live in the application. It works perfectly in development mode, but has strange behaviour when built and deployed to production. It works as follows:

A tiny script running separately collects broadcast UDP packages and shoves them into a mongo collection
The Meteor application then publishes a subset of this collection so the client can use it
The client subscribes and live-updates its view

The problem here is that the subscription appears to only get data about every 10 seconds, while these UDP packages arrive and gets shoved into the database several times per second. This makes the application behave weird

It is most noticeable on the collection of UDP messages, but not limited to it. It happens with every collection which is subscribed to, even those not populated by the external script

Querying the database directly, either through the mongo shell or through the application, shows that the documents are indeed added and updated as they are supposed to. The publication just fails to notice and appears to default to querying on a 10 second interval

Meteor uses oplog tailing on the MongoDB to find out when documents are added/updated/removed and update the publications based on this

Anyone with a bit more Mongo experience than me who might have a clue about what the problem is?

For reference, this is the dead simple publication function

/**
 * Publishes a custom part of the collection. See {@link  https://docs.meteor.com/api/collections.html#Mongo-Collection-find} for args
 *
 * @returns {Mongo.Cursor}      A cursor to the collection
 *
 * @private
 */
function custom(selector = {}, options = {}) {
        return udps.find(selector, options);
}

and the code subscribing to it:

Tracker.autorun(() => {
        // Params for the subscription
        const selector = {
                "receivedOn.port": port
        };
        const options = {
                limit,
                sort: {"receivedOn.date": -1},
                fields: {
                        "receivedOn.port": 1,
                        "receivedOn.date": 1
                }
        };

        // Make the subscription
        const subscription = Meteor.subscribe("udps", selector, options);

        // Get the messages
        const messages = udps.find(selector, options).fetch();

        doStuffWith(messages); // Not actual code. Just for demonstration
});

Versions:

Development:

node 8.9.3
mongo 3.2.15

Production:

node 8.6.0
mongo 3.4.10

Bmaed Riasbet · Accepted Answer

Meteor use two modes of operation to provide real time on top of mongodb that doesn’t have any built-in real time features. poll-and-diff and oplog-tailing

1 - Oplog-tailing

It works by reading the mongo database’s replication log that it uses to synchronize secondary databases (the ‘oplog’). This allows Meteor to deliver realtime updates across multiple hosts and scale horizontally. It's more complicated, and provides real-time updates across multiple servers.

2 - Poll and diff

The poll-and-diff driver works by repeatedly running your query (polling) and computing the difference between new and old results (diffing). The server will re-run the query every time another client on the same server does a write that could affect the results. It will also re-run periodically to pick up changes from other servers or external processes modifying the database. Thus poll-and-diff can deliver realtime results for clients connected to the same server, but it introduces noticeable lag for external writes. (the default is 10 seconds, and this is what you are experiencing , see attached image also ).

This may or may not be detrimental to the application UX, depending on the application (eg, bad for chat, fine for todos).

This approach is simple and and delivers easy to understand scaling characteristics. However, it does not scale well with lots of users and lots of data. Because each change causes all results to be refetched, CPU time and network bandwidth scale O(N²) with users. Meteor automatically de-duplicates identical queries, though, so if each user does the same query the results can be shared.

You can tune poll-and-diff by changing values of pollingIntervalMs and pollingThrottleMs.

You have to use disableOplog: true option to opt-out of oplog tailing on a per query basis.

Meteor.publish("udpsPub", function (selector) {
  return udps.find(selector, {
    disableOplog: true,
    pollingThrottleMs: 10000, 
    pollingIntervalMs: 10000
  });
});

Additional links:

https://medium.baqend.com/real-time-databases-explained-why-meteor-rethinkdb-parse-and-firebase-dont-scale-822ff87d2f87

https://blog.meteor.com/tuning-meteor-mongo-livedata-for-scalability-13fe9deb8908

How to use pollingThrottle and pollingInterval?

Meteor MongoDB subscription delivering data in 10 second intervals instead of live

The actual problem

Answers (2)

Related Questions