CODEWITHSUNDEEP
javascriptnode.jsmongodbexpressmongoose
tejaz
tejaz

Reputation: 67

Model.find({}) query using mongoose with node.js and express combination fails to fetch large amount of datasets of more than 200,000+ documents

I am using a simple get request using mongoose coupled with express and node to fetch all the documents in a particular collection I have defined in a MongoDB instance. It works fine for small amounts of data but is failing for large datasets. I am able to run the same query on Mongo Shell and after a decent amount of time, it is able to return the data.

I have tried to modify the query to use lean() function along with the find({}) function of mongoose but the problem still persists.

/* 
    Fetch all the players
    GET - /
*/
getPlayerRouter.route('/')
    .get((req, res, next) => {

        Player.find({}).lean()
            .then((players) => {
                res.status(200).json({
                    success: true,
                    totalPlayers: players.length,
                    players
                });
            })
            .catch((err) => console.log(err));

    });

I expect the query to fetch all the documents on the collection.

Upvotes: 0

Views: 58

Answers (1)

Vaibhav
Vaibhav

Reputation: 1499

var query=Player.find({}).stream();
query.on('data', (players)=> {
    res.status(200).json({
                      success: true,
                      totalPlayers: players.length,
                      players
                  });
 }).on('error',(err)=>{

 }).on('close',()=>{
    console.log('connection closed');
 });

You can use stream in mongoose to process large records.Comment on whether it works.

Upvotes: -1

Related Questions