CODEWITHSUNDEEP
mongodbmeteor
antanas_sepikas
antanas_sepikas

Reputation: 5714

Meteor,Mongo query find every nth document

I use time stamps in my collection so every document has a time stamp, user want's to get documents from "ts1" (time stamp 1) to "ts2" (time stamp 2), however there a too many documents in that interval so I wan't to return only every nth, for example if there is 100000 documents, I need to display 1000 documents, so 100000/1000=100. every 100th document.

Is this possible, and how could I achieve this.

PS. I need to query this inside Meteor publish method.

Here's what I'v got so far:

Meteor.publish('documents-chunk', function (from, to) {
    //get find documents count and get nth
    var count = Documents.find({time: {$gte: from, $lte: to}}).count();
    if (count > 2000) {
        var nth = Math.round(count / 1000);
        return Documents.find(/*query every nth*/);
    }
    return Documents.find({time: {$gte: from, $lte: to}});
});

SOLUTION:

I ~solved this problem using answer from Matt K.

This is what I'v done: first I modified my collection and added additional "id" field:

**

1.

**

Document.find({}, {sort: {time: 1}}).forEach(function (c, i) {
    Document.update(c, {$set: {id: i + 1}});
    console.log(i + 1);
});

This collection had little less than 1,5M records so it took some time, (also to note, I had to add index {time: 1} to this collection otherwise it would crash the database)

**

2.

**

Meteor.publish('documents-chunk', function (from, to) {
    var nth = Math.round(Documents.find({time: {$gte: from, $lte: to}}, {sort: {time: 1}}).count() / 1000);
    return Documents.find({time: {$gte: from, $lte: to, $mod: [nth, 0]}}, {sort: {time: 1}});
});

This worked for me, and now I get the result I needed;

I'v read at http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/ that this kinda approach is not recommended. But at this point of time I could not find any other solution to this problem, though I found that it is requested https://jira.mongodb.org/browse/SERVER-2397 so maybe in future there will be cleaner solution, but for now it works.

Upvotes: 4

Views: 1710

Answers (2)

Matt K
Matt K

Reputation: 4948

You can't, at least not to my knowledge. You've got three options:

publish and subscribe to all 100,000, then display every 1000th. Logically speaking, your query is based on the number of results returned from a query. That is a 2 step process no matter how you look at it.

If you wanted to be cute, you could have _id (or another field) be an auto-incrementing number. Then, set var qCount = cursor.count(). Then, query for _id % qCount === 0.

add a sample field to every 1000th record when it's created, then query for: {$exists: {sample: true}}

rethink the business logic. what's the value add of every 1000th record? If it's to "eyeball the data" you should probably be using an aggregate on the data anyways to get rid of outliers. (this is the right choice, but convincing the client is another story...)

Upvotes: 1

Michel Floyd
Michel Floyd

Reputation: 20246

If you believe that mongoDB _id values are truly randomly assigned, then you could simply order by _id and pick the first N of a set. This would give you N random values from the interval.

Meteor.publish('documents-chunk', function (from, to) {
  return Documents.find({time: {$gte: from, $lte: to}},{sort: {_id: 1}, {limit: 1000}});
});

I'd recommend running some statistics on the randomness of what you get back.

Upvotes: 1

Related Questions