CODEWITHSUNDEEP
node.jsmongodbexpressmongoose
Run
Run

Reputation: 57176

Mongoose-paginate: Overflow sort stage buffered data usage?

I am using Mongoose, Mongoose-paginate and ExpressJS. I have this error:

Error retrieving streams: MongoError: Runner error: Overflow sort stage buffered data usage of 34227161 bytes exceeds internal limit of 33554432 bytes

routes.js:

router.get("/", function(req, res, next) {

    var page = req.query.page === undefined ? 1 : req.query.page;
    var options = {
        page: page,
        limit: 30,
        sort: {
            created_at: 'desc'
        }
    };

    // https://github.com/edwardhotchkiss/mongoose-paginate
    Stream.paginate(query, options, function(err, result) {
        if (err) {
            console.log("Error retrieving streams: " + err);
            errorMessage = "A problem occurred retrieving the streams";
            return res.render("streams", {
                streams: {},
                errorMessage: errorMessage
            });
        }

        return res.render("streams/list", {
            streams: result.docs,
            page: parseInt(result.page),
            pages: parseInt(result.pages)
        });
    });
});

How can I solve this data limit issue in Mongoose?

EDIT:

Following this answer, I think the size in the data field is getting too large, so how can I index the data field?

This is my model:

model.js:

var mongoose = require("mongoose");
var mongoosePaginate = require('mongoose-paginate');

// Declare schema
var streamSchema = new mongoose.Schema({
    user_id: {
        type: String,
        required: true
    },
    title: {
        type: String,
        required: true
    },
    description: {
        type: String,
        required: true
    },
    hidden:{
        type: String
    },
    data: {
        type: Object
    }
});

streamSchema.plugin(mongoosePaginate);

// Export schema
// Model.paginate()
mongoose.model("Stream", streamSchema);

Can I ignore the data field when I query the list of the streams?

EDIT 2:

With this solution:

db.mydb.ensureIndex({created_at: 1})
{
    "createdCollectionAutomatically" : true,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}

I still get the same error...

EDIT 3:

Found the answer to exclude fields:

var options = {
    page: page,
    limit: 30,
    sort: {
        created_at: 'desc'
    },
    select: '-data'
};

I still get the same error...

Upvotes: 1

Views: 1166

Answers (3)

Run
Run

Reputation: 57176

Problem solved - must add index: 1 to the model:

var mongoose = require("mongoose");
var mongoosePaginate = require('mongoose-paginate');

// Declare schema
var streamSchema = new mongoose.Schema({
    title: {
        type: String,
        required: true
    },
    description: {
        type: String,
        required: true
    },
    hidden:{
        type: String
    },
    data: {
        type: Object
    },
    created_at: {
        type: Date,
        default: Date.now,
        index: 1
    },
});

streamSchema.plugin(mongoosePaginate);

// Export schema
// Model.paginate()
mongoose.model("Stream", streamSchema);

Then:

mongo> db.mydb.ensureIndex({created_at: 1})

Upvotes: 0

dyouberg
dyouberg

Reputation: 2332

You are running into the 32 MB limit of sorting in memory.

You want to create an index so MongoDB does the sorting for you.

Run the following command in your MongoDB Shell:

db.collectionName.ensureIndex({created_at: 1})

See here for more information: Overflow sort stage buffered data usage exceeds internal limit

Edit: To project out a field in your response you do it as follows:

db.collection.find({}, {field_to_see: 1, field_to_hide: 0})

Read here: MongoDB Explicit Exclusion

Upvotes: 3

Daphoque
Daphoque

Reputation: 4678

Use cursor to stream datas for big dataset :

var findCursor = Stream.find(query).cursor();
findCursor.on("data", function(data) {
  // datas...
});
findCursor.on("end", function(){
 // end of the stream
});

Upvotes: -1

Related Questions