CODEWITHSUNDEEP
javascriptnode.jsmongodbexpressmongoose
Jonathan Woollett-light
Jonathan Woollett-light

Reputation: 3243

Awaiting Seed Data Before Accepting Server Requests

As the titles suggest the issue here is a function in the server.js running after the base root has been loaded. Below you can see the function call and the root.

seedDB();
app.get("/",function(req,res)
{
    examBoard.find({}, function (err, examBoard)
    {
        console.log("examBoard.length: " + examBoard.length);
        res.render("landing", { examBoard: examBoard });
    });
});

The function does basic seeding of the database and thus must run before the base root. It outputs what you can see in the following image (most of the output is cut off).

enter image description here

The output in the red box is the output as result of the console.log in the base root. Here is the app.listen which is at the very bottom of the code, with everything above it.

app.listen(process.env.PORT, process.env.IP,function()
{
    console.log("Server started");
});

Here is the code for seedDB with full code including the arrays in this hastebin link (https://hastebin.com/acecofoqap.lua) (thought it would be a bit excessive to include them as they are rather large):

function seedDB() {
    user.remove({}, function (err) {
        if (err) {
            console.log("Could not remove user\n" + err);
        }
        else {
            console.log("Removed old user");
            examBoard.remove({}, function (err) {
                if (err) {
                    console.log("Could not remove examboards\n" + err);
                }
                else {
                    console.log("Removed old examboards");
                    question.remove({}, function (err) {
                        if (err) {
                            console.log("Could not remove questions\n" + err);
                        }
                        else {
                            console.log("Removed old questions");
                            user.register(new user
                                ({
                                    username: "admin",
                                    email: "[email protected]",
                                    role: "admin"
                                }),
                                "lu134r7n75q5psbzwgch", function (err, user) {
                                    if (err) {
                                        console.log("Failed to add admin\n" + err);
                                    }
                                    else {
                                        console.log("Admin added");
                                        examboardData.forEach(function (examSeed) {
                                            examBoard.create(examSeed, function (err, exam) {
                                                console.log("Creating new examboard");
                                                if (err) {
                                                    console.log("Could not create new examboard\n" + err);
                                                }
                                                else {
                                                    console.log("Created examboard");
                                                }
                                            });
                                        });
                                        var topicIncrementor = 0;
                                        questionData.forEach(function (questionSeed) {
                                            question.create(questionSeed, function (err, question) {
                                                if (err) {
                                                    console.log("Could not create new question\n" + err);
                                                }
                                                else {
                                                    console.log("Created question");
                                                    examBoard.find({}, function (err, exams) {
                                                        for (var i = 0; i < exams.length; i++) {
                                                            for (var t = 0; t < exams[i].modules.length; t++) {
                                                                for (var q = math.floor(topicIncrementor / 12); q < exams[i].modules[t].topics.length; q++) {
                                                                    exams[i].modules[t].topics[q].questions.push(question);
                                                                    topicIncrementor++;
                                                                }
                                                                topicIncrementor = 0;
                                                            }
                                                            exams[i].save();
                                                        }
                                                    });
                                                }
                                            });
                                        });
                                    }
                                });
                        }
                    });
                }
            });
        }
    });
}
module.exports = seedDB;

For my program to work here the seedDB function must run before the base root, if you can provide a solution or merely point me in the right direction it would be greatly appreciated.

Upvotes: 0

Views: 378

Answers (1)

Neil Lunn
Neil Lunn

Reputation: 151122

Bottom line is that your seedDB() needs either async callback or Promise resolution itself, and then only even start the 'http' part of the server when that operation is complete. Reason being is that the server then does not even accept requests until the data is confirmed to be loaded.

Implementing the Seed Function

With a modern release of nodejs, the simplest way to implement is by using async/await syntax

async function seedDB() {

  // Remove existing data
  await Promise.all(
    [user,examBoard,question].map( m => m.remove({}) )
  );

  // Create the user, wrap callback method with Promise
  await new Promise((resolve,reject) => {
    user.register( new user({
      username: "admin",
      email: "[email protected]",
      role: "admin"
    }),"lu134r7n75q5psbzwgch", (err, user) => {
      if (err) reject(err);
      resolve(user);
    });
  });

  // Create examBoard. .create() does actually accept an array.
  // Keep the results as well
  var exams = await examboard.create(examboadData);

  // Create questions. Same thing
  var questions = question.create(questionData);

  // Add questions to each exam topic

  for ( let question of questions ) { 
    for ( var i = 0; i < exams.length; i++ ) {
      for ( var t = 0; t < exams[i].modules.length; t++ ) {
        for ( var q = 0; q < exams[i].modules[t].topics.length; q++ ) {
          exams[i].modules[t].topics[q].questions.push(question);
        }
      }
      await exams[i].save();
    }
  }

}

Winding that back a little to see how it would look with plain Promise implementation:

function seedDB() {

  // Remove existing data
  return Promise.all(
    [user,examBoard,question].map( m => m.remove({}) )
  )
  .then( () =>
    // Create the user, wrap callback method with Promise
    new Promise((resolve,reject) => {
      user.register( new user({
        username: "admin",
        email: "[email protected]",
        role: "admin"
      }),"lu134r7n75q5psbzwgch", (err, user) => {
        if (err) reject(err);
        resolve(user);
      });
    })
  )
  .then( () => 
    Promise.all(
      // Create examBoard. .create() does actually accept an array.
      // Keep the results as well
      examboard.create(examboadData),

      // Create questions. Same thing
      question.create(questionData)
    )
  )
  .then( ([exams, questions]) => {
    // Add questions to each exam topic
    var items = [];

    for ( let question of questions ) { 
      for ( var i = 0; i < exams.length; i++ ) {
        for ( var t = 0; t < exams[i].modules.length; t++ ) {
          for ( var q = 0; q < exams[i].modules[t].topics.length; q++ ) {
            exams[i].modules[t].topics[q].questions.push(question);
          }
        }
        items.push(exams[i].save().exec());
      }
    }

    return Promise.all(items).then( () => Promise.resolve() );
  });

}

So it's basically the same thing except we "visibly chain" the promises rather than use the await sugar that can be used in modern environments.

The key point to understand here is that "everything in the mongoose API returns a Promise", and therefore there are much cleaner ways to implement awaiting completion and chaining of calls.

This goes for the simple chaining of .remove() calls:

  await Promise.all(
    [user,examBoard,question].map( m => m.remove({}) )
  );

Which is something I commonly write when "seeding" initial data by "cycling all registered models":

  await Promise.all(
    Object.keys(conn.models).map( m => conn.models[m].remove({}) )
  )

Or if there are methods that don't actually have promises, then you can always "wrap them" in a Promise, as is done with user.register on the presumption that it is "callback only". Actual implementation may differ, but this is a common method of wrapping a callback by demonstration.

At the end of the day, everything waits properly and the whole function resolves only when everything is complete. This allows you to move on to the important part.

Implementing Awaiting Server Start

The whole reason for making the seed function an "async" return is so we know when it's completed and will then only call the .listen() method when that is confirmed, so the server simply will not accept requests until the data is ready.

Again, depending on the node version available you either use async/await syntax or chain the promises to resolve.

So using async/await, you main server startup should look something like this:

const mongoose = require('mongoose'),
      app = require('express')();
      // and other module includes, including the seedDB source

const uri = 'mongodb://localhost/test',
      options = { useMongoClient: true };

mongoose.Promise = global.Promise;
mongoose.set('debug',true); // useful before deployment to see requests

// Possibly other express handler set up

// Root handler
app.get('/', function(req,res) {
  examBoard.find({}, function (err, examBoard) {
    console.log("examBoard.length: " + examBoard.length);
    res.render("landing", { examBoard: examBoard });
  });
});

// Main startup
(async function() {
  try {

    const conn = await mongoose.connect(uri,options);

    // Seed and await
    await seedDB();

    // Then we listen
    await app.listen(5858);

    console.log('server listening');
  } catch(e) {
    console.error(e);
    mongoose.disconnect();  // Probably not realistically
  }
})();

The async/await suguar here allows you to simply list each action sequentially, and also within a try/catch block which would trap any errors without all the other callback mess.

Alternately a "chaining" approach would be:

const mongoose = require('mongoose'),
      app = require('express')();
      // and other module includes, including the seedDB source

const uri = 'mongodb://localhost/test',
      options = { useMongoClient: true };

mongoose.Promise = global.Promise;
mongoose.set('debug',true); // useful before deployment to see requests

// Possibly other express handler set up

// Root handler
app.get('/', function(req,res) {
  examBoard.find({}, function (err, examBoard) {
    console.log("examBoard.length: " + examBoard.length);
    res.render("landing", { examBoard: examBoard });
  });
});

// Main startup
mongoose.connect(uri,options)
  .then( () => seedDB() )
  .then( () => app.listen(5858) )
  .then( () => console.log('server listening') )
  .catch( e => console.error(e) );

Which really only differs in that aside from the "chaining" we use .catch() on the end of the chain.

Note that in all cases the promises here are "fail-fast" here, in that any error is going to essentially either be caught in that try/catch or .catch() respectively, without attempting to continue. In most cases, that's what you want, but you alternately handle it by either using similar blocks or .catch() handlers at finer grained areas.

Design Notes

So the question as posed had some pretty messy code with a lot of callback nesting that the more modern features demonstrated here are meant to "clean up" and make things functional and readable again. You should immediately note the significant difference in comprehension.

As I see it, there are still problems. The main case in point being the "all questions being added to everything", which may well be okay for testing purposes but I doubt this was your intent. And there are likely a lot more efficient ways of joining up those questions, but it's basically off the main topic of the question which was simply about "awaiting the async completion".

As such there are some changes in parts of the code, but only in areas where the implemented code actually was not doing anything, or at least not doing anything that you may have expected. ( topicIncrementor is only ever exposed to the loop initialize as 0, and will never be different ).

Overall this should give you a reasonable guide to understanding how things can be done a lot differently as well as see how clean and simple these tasks can be, so that they read exactly like what they are designed to do, instead of looking more like "functional noodles".

Upvotes: 1

Related Questions