Meteor publishing and subscribing to a large Collection

Question

So let's take this scenario, in an e-commerce application, a user searches for "wrist watches".

Is it advisable for me to publish and subscribe the entire Products collection ? Because that table my grow a lot in size. Is it possible for me to fetch from a collection without subscribing to it ?

Also, in Meteor 1.3, which is the best place to define collections ? From what I read, it has to be in /imports/api, but some light on it might be helpful.

Thanks,

Answer 1

When you want to get data to your meteor client, you have three options - choose your own adventure.

option 1: publish the whole collection

pros: easy to implement, fast to use/filter on the client once the data has arrived, publication can be reused on the server for all clients

cons: doesn't scale well / doesn't work past a couple of thousand documents, may be a lot to transmit to the client

use when: you have a small size-bounded collection and the client needs all of it for filtering / searching / selecting

option 2: use a method

You can have a meteor method deliver the filtered documents to the client instead of publishing them. I.e. the user searches for "wrist watches", and the method delivers only those documents. See this section of the guide for more details. You can stuff the documents into a local collection if you like, but it isn't required.

pros: performance, scalability, data isolation (you don't have to worry that some subset of the documents were added by another subscription)

cons: it's more work to set up and manage than a subscription

use when: you have an unbounded collection and you need a subset in the most performant way

option 3: use a reactive subscription

This is very similar to (2) except you'll be re-subscribing in an autorun after changing your search parameters. See this section for more details.

pros: easier to implement than (2)

cons: more computationally expensive an a bit slower than (2) with the possible exception that publications could be reused on the server (unlikely in the case of a search)

use when: you have an unbounded collection and you need a subset with the least amount of effort/code

---

Without knowing more about your particular use case, I'd go with (2).

As for where to define your collections, see this section and the todos app for examples. The recommendation is to use imports/api as you mentioned. For an explanation of why, see this question. If you need more detail, I'd recommend opening a separate question.

Answer 2

Generally speaking we don't post all fetched data onto a page at once. It too lengthy for the customers in terms of user experience. A common advice is pagination plus sorting.

As to Meteor, collections on the server are different from collections on the client. In short, a collection on the client is a subset of the server collection. Data in that subset is determined by a publication-subscription mechanism of Meteor. Data is published on the server and you subscribe to it on the client. This way you derive the subset. Morever you can define filtering, sorting, count, ect to shape the derived subset based on what and how you like the subset to be used on the client. The documentation contains a pretty decent guide and details about Meteor collections.

The place to define collections is really flexible in Meteor. It doesn't have to be /imports/api. It can be any location that can be accessed by both the server and the client, because in general use cases the server needs to see the data and define methods for manipulating the collection, and the client needs to see it as well for rendering data on web pages. But, as said, it is flexible and depends on how you implement and structure your applications. It can a location accessible by both the server and the client, but it needs not to be. In some cases the collections are defined on the server only, and the client fetch the data from implicit and indirect protocols. Meteor method is one of them, and Restful API is another to name a few. It's case by case and you do what you feel best. That is where the fun is from. Subscription is common and convenient but not the only.

Meteor defines special rules to folder access on the server and client respectively, and Meteor 1.3 imposes a new rule for modulation. I enjoy reading Meteor documentation and find them really useful, like this one helps develop solid knowledge on the afore-mentioned rules.