Mongoose and population nested

Question

I have read something about Mongoose like it is not able to manage nested population.

The fact is that when I do a findOne() call, the object returned is able to use the populate function. But it doesnt seem to work...

Can you help me processing a 2 level populate ? Multiple query are not a probleme.

Here's my CoffeeScript code :

Agency.findOne({"agencyUsers": idUser}, '_id agencyUsers agencySites agencyDevices').populate([
      {path: 'agencyUsers', match: { actif: true}, select: '_id userPin'},
      {path: 'agencySites', match: { actif: true}, select: '_id siteName siteBuildings siteSectors siteUsers'},
      {path: 'agencyDevices', match: { actif: true}, select: '_id deviceMac'}
    ])
    .exec((err, res)=>
      if err
        deffered.reject err
      else
        res.populate('agencySites.siteBuildings', (errs, ress)=>
          deffered.resolve(res)
        )
    )

EDIT :

According to the fact that we need to populate buildings in sites (in agency)

AgencySchema :

{
  agencyName: String,
  agencyStreet: String,
  agencyCity: String,
  agencyPostal: String,
  agencyCountry: String,
  agencyPhone: String,
  agencyFax: String,
  agencyMail: String,
  agencySiret: String,
  agencyAgencies: [
    { type: Schema.ObjectId, ref: 'Agency' }
  ],
  agencyDevices: [
    { type: Schema.ObjectId, ref: 'Device' }
  ],
  agencySites: [
    { type: Schema.ObjectId, ref: 'Site' }
  ],
  agencyUsers: [
    { type: Schema.ObjectId, ref: 'User' }
  ],
  agencyModules: [
    { type: Schema.ObjectId, ref: 'Module' }
  ],
  actif: {type: Boolean, default: true},
  actif_date: { type: Date, default: null },
  creation_date: { type: Date, default: Date.now },
  edit_date: { type: Date, default: null }
}

Site schéma :

{
  siteName: String,
  siteStreet: String,
  siteCity: String,
  sitePostal: String,
  siteCountry: String,
  sitePhone: String,
  siteMail: String,
  siteAgencies: [
    { type: Schema.ObjectId, ref: 'Agency' }
  ],
  siteUsers: [
    { type: Schema.ObjectId, ref: 'User' }
  ],
  siteSectors: [
    { type: Schema.ObjectId, ref: 'Sector' }
  ],
  siteBuildings: [
    { type: Schema.ObjectId, ref: 'Building' }
  ],
  siteModules: [
    { type: Schema.ObjectId, ref: 'Module' }
  ],
  actif: {type: Boolean, default: true},
  actif_date: { type: Date, default: null },
  creation_date: { type: Date, default: Date.now },
  edit_date: { type: Date, default: null }
}

The building schema :

{
  buildingName: String,
  buildingFloors: [
    { type: Schema.ObjectId, ref: 'Floor' }
  ],
  actif: {type: Boolean, default: true},
  actif_date: { type: Date, default: null },
  creation_date: { type: Date, default: Date.now },
  edit_date: { type: Date, default: null }
}

Answer 1

Since version ^4.1.x, deep population is actually a thing in mongoose, so doing something like this is valid :-

Post.find({}).populate({
   path: 'user',
   select: 'id name',
   populate: {
      path: 'friends',
      select: 'id name,'
      populate: {
          path: 'friends',
          select: 'id name',
      }
   }
})

populate can take either an object or an array of objects as a value, which means you can populate sibling attributes that reference different collections. for more information, check this post: http://frontendcollisionblog.com/mongodb/2016/01/24/mongoose-populate.html

Answer 2

@Neil Lunn's answer is correct, I just wanted to shine some light on co incase you were interested in cleaning up your code a bit:

co(function* {
  var agency = yield Agency.findOne({'agencyUsers': idUser}, '_id agencyUsers agencySites agencyDevices')
    .populate(
      { 'path': 'agencyUsers', 'match': { 'actif': true }, 'select': '_id userPin' },
      { 'path': 'agencySites', 'match': { 'actif': true }, 'select': '_id siteName siteBuildings siteSectors siteUsers' },
      { 'path': 'agencyDevices', 'match': { 'actif': true }, 'select': '_id deviceMac' })

  agency = yield Agency.populate(agency, 'agencyAgencies.siteAgencies')
  agency = yield User.populate(agency, 'agencyAgencies.siteUsers')
  agency = yield Sector.populate(agency, 'agencyAgencies.siteSectors')
  agency = yield Building.populate(agency, 'agencyAgencies.siteBuildings')
  agency = yield Module.populate(agency, 'agencyAgencies.siteModules')

  return agency
})
.then(agency => {
   Floor.populate(agency, 'agencyAgencies.siteBuildings.buildingFloors', function (err, res) {
     if (err)
       deferred.reject(err)

     deferred.resolve(agency)
   })
})
.catch(deferred.reject)

Answer 3

JavaScript response only. You do the translation :)

This doesn't really work in this context for actual reasons that escape me at the moment, but in order to do what you want you need to call the "Model" form of .populate() instead:

Agency.findOne({"agencyUsers": idUser}, '_id agencyUsers agencySites agencyDevices')
    .populate([
      { "path": "agencyUsers", "match": { "actif": true}, "select": "_id userPin" },
      {
        "path": "agencySites", 
        "match": { "actif": true }, 
        "select": "_id siteName siteBuildings siteSectors siteUsers"
      },
      {
        "path": "agencyDevices", 
        "match": { "actif": true}, 
        "select": "_id deviceMac"
      }
]).exec(function(err,doc) {
    if (err) {
        deffered.reject(err);
    } else {
        async.waterfall([
           function(callback) {
               Agency.populate( doc, { "path": "agencyAgencies.siteAgencies" },callback);
           },
           function(doc,callback) {
               User.populate( doc, { "path": "agencyAgencies.siteUsers" },callback);
           },
           function(doc,callback) {
               Sector.populate( doc, { "path": "agencyAgencies.siteSectors" },callback);
           },
           function(doc,callback) {
               Building.populate( doc, { "path": "agencyAgencies.siteBuildings" },callback);
           },
           function(doc,callback) {
               Module.populate( doc, { "path": { "agencyAgencies.siteModules" },callback);
           }
        ],function(err,res) {
           if (err) {
               deffered.reject(err);
           } else {
               Floor.populate(res,{ "path": "agencyAgencies.siteBuildings.buildingFloors" },function(err,res) {
                   if (err) {
                       deffered.reject(err);
                   } else {
                       deffered.resolve(res);
                   }
               });
           }
        });
    }
});

My use of "async.waterfall" notwithstanding, just trying to avoid "indentation creep" by each embedded iteration.

So as you can see it is actually necessary to call .populate() in this way for each specific model type. Also there was the necessary call inside the "again nested" "Building" model reference also needs to be called after "that" document is populated. So it is that "call stack" that is part of the nested populate problem you mentioned earlier.

With such heavy referencing you might do well to consider schema redesigns and/or embedding a lot of this data. It looks very "relational" and as such you probably are not getting the full MongoDB benefit by doing it this way.