How to synchronize requests?

Question

I am using nodejs+express+mongoose. Assuming I have 2 schemas and models in place: "Fruits" and "Vegetables".

Assuming I have the following:

var testlist = ["Tomato", "Carrot", "Orange"];
var convertedList = [];
// Assume res is the "response" object in express

I wan to be able to check each item in the array against the "fruits" and "vegetables" collections respectively and insert them into a converted list where Tomato, Carrot, and Broccoli are replaced with their respective documents.

Below I have some pseudocode of what I think it would be, but know not how to do this.

for(var i = 0; i < testlist.length; i++) {
var fruitfind = Fruit.find({"name":testlist[i]});
var vegfind = Vegetables.find({"name":testlist[i]});

// If fruit only
if(fruitfind) {
convertedList.push(fruitfindresults);
} 
// If vegetable only
else if(vegfind) {

convertedList.push(vegfindresults);
} 
// If identified as a fruit and a vegetable (assume tomato is a doc listed under both fruit and vegetable collections)
else if (fruitfind && vegfind) {
convertedList.push(vegfindresults);
}
}

// Converted List should now contain the appropriate docs found.
res.send(convertedList) // Always appears to return empty array... how to deal with waiting for all the callbacks to finish for the fruitfind and vegfinds?

What is the best way to do this? Or is this even possible?

Answer 1

Assuming there's only one of each fruit/vegetable and that you intended to push a veggie that's found in both collections twice.

var async = require("async"),
    testlist = ["Tomato", "Carrot", "Orange"];

async.map(testlist, function (plant, next) {
  async.parallel([function (done) {
    Fruit.findOne({"name": plant}, done);
  },
  function (done) {
    Vegetables.findOne({"name": plant}, done);
  }], function (err, plants) { // Edited: before it was (err, fruit, veggie) which is wrong
    next(err, plants);
  });
},
function (err, result) {
  var convertedList = [].concat(result);
  res.send(convertedList);
});

Note: haven't actually tested the code, but it should work. The async module is excellent for managing callbacks like this btw.

Update

To get each fruit only once, the async.parallel callback simply have to be rewritten like this:

function (err, plants) {
  next(err, plants[0] || plants[1]);
}

And there's no concat needed anymore in the .map callback:

function (err, result) {
  res.send(result);
}

Answer 2

find is an asynchronous function, and it makes a request to the mongo database. This means two things:

1. The functions will not return results immediately. The find function in mongoose follows a very common async pattern. It accepts a "callback" function which it will call with either the results, or an error. By node convention, if the first argument is not null, it is an error.

   // So typically you'd call find like this
   SomeModel.find({your: 'conditions'}, function (err, results) { 
     if (err) {
       // some error has occurred which you must handle
     } else {
       res.send(results);
     }
   })
   // note that if code existed on the lines following the find, it would
   // be executed *before* the find completed.
   

2. As every query is firing another request off to the database, you typically want to limit the number if you can. In this case, instead of finding each fruit/veg by name, you could look for all the names at once by using mongo's $in.

With these two things in mind, your code might look something like this:

// here *first* we're finding fruits
Fruit.find({name: {$in: testlist}}, function (err, fruits) {

  // when the fruit request calls back with results, we find vegetables
  Vegetable.find({name: {$in: testlist}}, function (err, vegetables) {

    // finally concat and send the results
    res.send(fruits.concat(vegetables));
  });
});

To have both requests happen in parallel, a little more work is required. You could use a library like async, or write something yourself like:

var fruits
  , vegetables
  , done = function () {
      if (fruits && vegetables) {
        res.send(fruits.concat(vegetables));
      }
    }

Fruit.find({name: {$in: testlist}}, function (err, docs) {
  fruits = docs;
  done();
});

Vegetable.find({name: {$in: testlist}}, function (err, docs) {
  vegetables = docs;
  done();
});

Note that both examples here simply concat the results and send them, as it's not clear how you want the results processed. This means that if a tomato, for example, was in both lists, it would appear in the results twice, both the Vegetable and Fruit documents.

You'll also need to handle any errors coming back from mongoose.

Edit: Uniquely named docs

In light of your comment, this is one way you might return only one doc for Tomato (or other records that are both fruit and vegetable)

// after retrieving fruits and vegetables, create a map which will
// serve to weed out docs with duplicate names
var map = {};

fruits.forEach(function (fruit) {
  map[fruit.name] = fruit;
});

vegetables.forEach(function (vegetable) {
  map[vegetable.name] = vegetable;
});

var results = [];

// this would sort by name
Object.keys(map).sort().forEach(function (key, i) {
  results[i] = map[key];
});

res.send(results);

Note that this sort of thing becomes much more complicated if you need to sort and paginate or otherwise limit the result of the two queries, and if you need that you might rather consider keeping the documents in the same collection.