CODEWITHSUNDEEP
meteor
Roman
Roman

Reputation: 215

publish/subscription with big data Collection

I have a very large collection (~40000 documents with ~20-25 fields including array-fields with a set of ~500 items in) and ~2000 subscribers (they are just bots now).

So it's really hard work on client when user subscribe on whole collection (excluding some fields in server's publish) and use Collection.find with custom filters, sort and order

I tryed to use publish with options: i.e. client-defined filters & etc. But in this case I have too many memory leaks on server and epic fail :) after a few hours.

Can anyone advise some publish-subscribe schema for that kind of collections? I do not ask for clear solution but some useful thoughts

Thanks

Upvotes: 6

Views: 1999

Answers (2)

Scott Speidel
Scott Speidel

Reputation: 21

You might want to try https://github.com/alethes/meteor-pages

It has:

Incremental subscriptions. Downloads only what's needed, not the entire collection at once. Suitable for large datasets.

Upvotes: 2

Tomas Hromnik
Tomas Hromnik

Reputation: 2200

Do NOT publish all documents. Publish just documents you want to show and use pagination. I wanted to write a tutorial about pagination in Meteor but I don't have much time for it now.

Use Iron Router and its waitOn function for client subscribtions.

Router.map(function () {
   this.route('postShow', {
      path: '/posts/:page',

      before: function() {
         Session.set('page', this.params.page);
      },

      waitOn: function () {
         //how many documents we want to per page
         var perPage = 50;
         return Meteor.subscribe('posts', {}, {
           //limit
           limit: perPage,
           //we need to compute how many documents to skip
           skip:  perPage * Session.get('page') - perPage
         });
      },

      data: function() {
         return {
            //variable posts available for Handlebars: {{#each posts}}...
            posts: Posts.find()
         }
      } 
   });
});

And Publish:

Meteor.publish('posts', function(params, opts) {
   return Posts.find(params, opts);
});

Upvotes: 0

Related Questions