Can Meteor correctly handle data updated externally directly to the MongoDB database?

Question

Context: I already have a Nodejs server app that takes care of reading and writing to a MongoDB database autonomously. This service doesn't have any public endpoint, so the only way to communicate with it is through the database itself.

Now I want to build an independent web application, a website that will consume the data available at the previously mentioned MongoDB database. Most of this data should be shown in real-time to the user. After some framework research, I'm considering to use Meteor, but I couldn't find any useful info on whether it integrates well with pre-existing services.

So the actual question is: Can I benefit from Meteor's framework, particularly from all the real-time client data updates, while directly and externally using the MongoDB database?

For example, if I'm showing a table of "players" in my site, and a new player is added or the score of a player is updated externally and directly in the underlying MongoDB database, will Meteor be able to update the score in all the clients that have the site open, or will this requirement make Meteor unsuitable for my project?

Answer 1

Can I benefit from Meteor's framework, particularly from all the real-time client data updates, while directly and externally using the MongoDB database?

Meteor is configured to connect to an external mongo database. You have mongo packaged on a local development app but that is just for comfort and easy integration.

In production you will always have to connect Meteor to your running Mongo instance using the MONGO_URL environment variable.

Bonus: you can set the MONGO_URL even in dev mode to connect to a specific db in dev mode. Be aware that you can CRUD everything on this db, use with care.

Under the hood Meteor uses the node mongo driver. You are basically able to use all the API of this driver (see this post for details on how to call native Mongo methods)

For example, if I'm showing a table of "players" in my site, and a new player is added or the score of a player is updated externally and directly in the underlying MongoDB database, will Meteor be able to update the score in all the clients that have the site open, or will this requirement make Meteor unsuitable for my project?

With Meteor's publication / subscription system you basically control what data is published to the client. Publications are running each time the data changes (similar to observer pattern).

If all your clients subscribe to the data of a collection they will get the updates, once the collection updates. And now coming to your most wanted feature: this also works if this data is updated by some external source.

It also works with very rapid update intervals. I worked on a recent project where massive amounts of data have been updated via python on a Mongo-DB collection. The Meteor app listened to it and displayed the data to the clients in "realtime" (or what users perceive as real time).

However, there are some pitfalls and it is good to know them in advance.

1. Meteor creates documents with an _id of type string and not Mongo.ObjectID. However it is capable of reading it and writing it if you use it correctly.

2. Meteor wraps collections with an own API that integrates the collection best with its fibers based environment. If you need to use functionality beyond please read here and here.

3. Returned cursors behave slightly different but as with Collections there are also all native functionalities available if you receive the cursor from a rawCollection

4. Double check the data types you use on your collections. For example stick with the same Date types (like ISODate) so there are no unintended errors afeter an update. There is also a Meteor counterpart to mongoose named simpl-schema (npm package) which is a good way to keep structure on your collections.

Summarized, if you consider the most of the Meteor guide and API docs you should be on a good track as the integration of externally updated collections usually runs very well and it is mostly about understanding the pub/sub system to make it run.

---

Edit:

However and considering pitfall number 1, if Meteor uses custom ids, will it really be able to handle new documents added to a collection externally (for example a new player), if the id of this new document is set externally?

Yes it does but you need to be aware of a different query. If the (externally created) doc has the following value:

{ "_id" : ObjectId("4ecc05e55dd98a436ddcc47c") }

Then you have to change your (Meteor) query from

collection.findOne("4ecc05e55dd98a436ddcc47c") // returns undefined

to

collection.findOne({ "_id" : new Mongo.ObjectID("4ecc05e55dd98a436ddcc47c") }) // returns { _id: ObjectID { _str: '4ecc05e55dd98a436ddcc47c' } }

And indeed, will I be able to instruct Meteor to create the documents with custom IDs that are consistent with the IDs already created by my external service? Or will I need to make my external app use IDs similar to Meteor's (i.e. strings)?

Same pattern works for creating documents. Instead of

collection.insert({ foo:'bar' })

you pass a newly created ObjectID:

collection.insert({ foo:'bar', _id: new Mongo.ObjectID() })

And talking about potential pitfalls, does Meteor store data in any unusual way? I mean, does it add any collection to the DB or any special field to documents that are inserted or alike to keep track of these?

Or can I expect totally normal and simple documents apart from the custom _id field which is simply a string?

If you create documents in Meteor with the method above, you should not need to worry about _id being a String.

Besides that documents are the way they should be (see the data format bridge). However, if there is an exception you found that has not been mentioned here, feel free to comment as this may be important to other users as well.