Unable to add new properties in nested callback while joining MongoDB collections in Node.js

Question

I'm trying to implement a JOIN-like operation between two MongoDB collections in Node.js/Express: persons and addresses. Each person has an id_adr field referencing their address. Here's my current code:

var ret;
app.get('/tp/show/method', function (req, res) {
    ret = {};
    User2Model.find(function(err, users) {
        if(err) {
            console.error('Error finding users: ' + err);
        }
        var last_id;
        for(var user_id in users) {
            last_id = users[user_id]._id;
            ret[last_id] = users[user_id];
        }
        
        for(var user_id in users) {
            AdrModel.find({ 'user_id': users[user_id]['id_adr'] }, function(err, adr) {
                if (err) {
                    console.error('Error finding addresses: ' + err);
                }
                for(var i in adr) {
                    for(var user_id in users) {
                        if (users[user_id].id_adr == adr[i].user_id) {
                            // Trying to add address information to the user object
                            ret[users[user_id]._id] = adr;
                            
                            if(users[user_id]._id == last_id) {
                                var url_parts = url.parse(req.url, true);
                                var query = url_parts.query;
                                res.setHeader('content-type', 'application/javascript');
                                json = query.callback + '(' + JSON.stringify({
                                    data: {success: 1, value: ret}
                                }) + ')';
                                res.send(json);
                            }
                            break;
                        }
                    }
                }
            });
        }
    });
});

Problem: Even though ret is defined in the outer scope, I'm unable to add new properties to the objects within it from the nested callback. When I try to add address information, it only works when overwriting existing properties, but not when trying to add new ones like "addr".

Expected Behavior: I want to be able to add the address information as a new property to each user object in the ret variable.

Question: What am I doing wrong, and how can I properly add new properties to the objects within ret from inside the nested callback?

Additional Context:

• Using Node.js with Express
• MongoDB with Mongoose
• Implementing JSONP response

Answer 1

Looking at the code, I found the issues:

• Callbacks were causing race conditions with the shared ret object
• Inefficient nested loops making multiple separate DB queries
• No proper error handling
• Hard to track when all async operations complete

    const express = require('express');
    const app = express();
    
    // Define the route handler as async
    app.get('/tp/show/method', async (req, res) => {
        try {
            // Use lean() for better performance since we don't need Mongoose documents
            const users = await User2Model.find().lean().exec();
            
            // Get all unique address IDs
            const addressIds = [...new Set(users.map(user => user.id_adr))];
            
            // Fetch all addresses in one query
            const addresses = await AdrModel.find({
                'user_id': { $in: addressIds }
            }).lean().exec();
            
            // Create a map of addresses for O(1) lookup
            const addressMap = new Map(
                addresses.map(addr => [addr.user_id, addr])
            );
            
            // Combine user and address data
            const result = users.reduce((acc, user) => {
                const userAddress = addressMap.get(user.id_adr);
                acc[user._id] = {
                    ...user,
                    address: userAddress || null
                };
                return acc;
            }, {});
    
            // Handle JSONP response
            const callback = req.query.callback;
            res.setHeader('content-type', 'application/javascript');
            const jsonResponse = `${callback}(${JSON.stringify({
                data: { 
                    success: 1, 
                    value: result 
                }
            })})`;
            
            res.send(jsonResponse);
    
        } catch (error) {
            console.error('Error:', error);
            res.status(500).send(`${req.query.callback}({
                data: { 
                    success: 0, 
                    error: "Internal server error" 
                }
            })`);
        }
    });

Hope this helps!

Answer 2

As I indicated in comments to @Tomalak's solution above, events can also be used to manage program flow in an async environment.

Here's an alternate partial solution that uses just that approach.

Please note that of the various ways of accomplishing this goal (I know of at least three, or four if you accept that the pain of "Callback Hell" can be ameliorated through the use of callbacks defined outside of and only referenced inline by their caller), I prefer using events since they are a more natural way for me to think about this class of problem.

Take-aways

1. Events are an efficient and easily understandable way to manage program flow in an async programming environment.
2. Rather than simple triggers, events can be used transport any data so they can be used further on for any purpose.
3. Events can easily call other events without worrying about scope.
4. Event processing allows you to unwind your code such that it becomes easier to track, and thus debug, as well as reducing the burden on the stack typically seen in deeply nested or recursive code. In other words, events are fast and very memory efficient.

Explanation

The code first defines two mocks:

1. an App class which provides a get method, allowing us to mock out the OP's app instance, and
2. a User2Model singleton that provides a find function for the same purpose.

It then documents the following events:

1. error - which is called on any errors to print a message to console and exit the program
2. get - which is fired with the result of the app.get method and immediately fires the processUsers event with {req:req,res:res}
3. processUsers - fired by the get event handler with a mocked array of user objects, sets up a results object and a last_id value, and then calls the nextUser event.
4. nextUser - fired by the processUsers event which picks the next user off the users array, sets evt.last_id, adds the user to the evt.results, and emits itself, or if there are no users left on the evt.users array, emits complete
5. complete - fired by nextUser and simply prints a message to console.

Event handlers are next defined using the 'on'+eventName convention.

And finally, we

1. define an eventHandlers object, to map handlers to their appropriate events,
2. instantiate our app instance, and
3. invoke its get method with a callback that simply emits a get event to start the ball rolling.

I've documented most of the solution using jsdoc and added logging messages to show progress as each event is emitted and its handler invoked. The result of the run is included after the code. (The http req and res objects have been commented out of the log messages for the sake of brevity.)

One final note, while this example is 269 lines long, most of it is documentation.

The actual code (without the mocks) is only about 20 or 25 lines.

Code

/*

 Example of using events to orchestrate program flow in an async
 environment.

 */

var util = require('util'),
    EventEmitter = require('events').EventEmitter;

// mocks

/**
 * Class for app object (MOCK)
 * @constructor
 * @augments EventEmitter
 */
var App = function (handlers) {
  EventEmitter.call(this);
  this.init(handlers);
};
util.inherits(App, EventEmitter);

/**
 * Inits instance by setting event handlers
 *
 * @param {object} handlers
 * @returns {App}
 */
App.prototype.init = function (handlers) {
  var self = this;
  // set event handlers
  Object.keys(handlers).forEach(function (name) {
    self.on(name, handlers[name]);
  });
  return self;
};

/**
 * Invokes callback with req and res
 * @param uri
 * @param {App~getCallback} cb
 */
App.prototype.get = function (uri, cb) {

  console.log('in app.get');

  var req = {uri: uri},
      res = {uri: uri};
  /**
   * @callback App~getCallback
   * @param {object} req - http request
   * @param {object} res - http response
   * @fires {App#event:get}
   */
  cb(req, res);
};

/**
 * Data access adapter - (MOCK)
 * @type {object}
 */
var User2Model = {};
/**
 *
 * @param {User2Model~findCallback} cb
 */
User2Model.find = function (cb) {
  var err = null,
      users = [
        {_id: 1},
        {_id: 2}
      ];
  /**
   * @callback User2Model~findCallback
   * @param {Error} err
   * @param {Array} users
   */
  cb(err, users);
};


// events

/**
 * Error event.
 *
 * @event App#error
 * @type {object}
 * @property {object} [req] - http request
 * @property {object} [res] - http response
 * @property {string} where - name of the function in which the error occurred
 * @property {Error} err - the error object
 */

/**
 * Get event - called with the result of app.get
 *
 * @event App#get
 * @type {object}
 * @property {object} req - http request
 * @property {object} res - http response
 */

/**
 * ProcessUsers event - called
 *
 * @event App#processUsers
 * @type {object}
 * @property {object} req - http request
 * @property {object} res - http response
 * @property {Array} users - users
 */

/**
 * NextUser event.
 *
 * @event App#nextUser
 * @type {object}
 * @property {object} req - http request
 * @property {object} res - http response
 * @property {Array} users
 * @property {*} last_id
 * @property {object} result
 */

/**
 * Complete event.
 *
 * @event App#complete
 * @type {object}
 * @property {object} req - http request
 * @property {object} res - http response
 * @property {Array} users
 * @property {*} last_id
 * @property {object} result
 */

// event handlers

/**
 * Generic error handler
 *
 * @param {App#event:error} evt
 *
 * @listens App#error
 */
var onError = function (evt) {
  console.error('program error in %s: %s', evt.where, evt.err);
  process.exit(-1);
};

/**
 * Event handler called with result of app.get
 *
 * @param {App#event:get} evt - the event object
 *
 * @listens App#appGet
 * @fires App#error
 * @fires App#processUsers
 */
var onGet = function (evt) {
  console.log('in onGet');
  var self = this;
  User2Model.find(function (err, users) {
    if (err) {
      console.log('\tonGet emits an error');
      return self.emit('error', {
        res:evt.res,
        req:evt.req,
        where: 'User2Model.find',
        err: err
      });
    }
    self.emit('processUsers', {
      //req:req,
      //res:res,
      users: users
    });
  });
};

/**
 * Handler called to process users array returned from User2Model.find
 *
 * @param {App#event:processUsers} evt - event object
 * @property {object} req - http request
 * @property {object} res - http response
 * @property {Array} users - array of Users
 *
 * @listens {App#event:processUsers}
 * @fires {App#event:nextUser}
 */
var onProcessUsers = function (evt) {
  console.log('in onProcessUsers: %s', util.inspect(evt));
  var self = this;
  evt.last_id = null;
  evt.result = {};
  self.emit('nextUser', evt);
};

/**
 * Handler called to process a single user
 *
 * @param evt
 * @property {Array} users
 * @property {*} last_id
 * @property {object} result
 *
 * @listens {App#event:nextUser}
 * @emits {App#event:nextUser}
 * @emits {App#event:complete}
 */
var onNextUser = function (evt) {
  var self = this;

  console.log('in onNextUser: %s', util.inspect(evt));

  if (!(Array.isArray(evt.users) && evt.users.length > 0)) {
    return self.emit('complete', evt);
  }

  var user = evt.users.shift();

  evt.last_id = user._id;

  evt.result[evt.last_id] = user;

  self.emit('nextUser', evt);
};

/**
 * Handler invoked when processing is complete.
 *
 * @param evt
 * @property {Array} users
 * @property {*} last_id
 * @property {object} result
 */
var onComplete = function (evt) {
  console.log('in onComplete: %s', util.inspect(evt));
};

// main entry point

var eventHandlers = { // map our handlers to events
  error: onError,
  get: onGet,
  processUsers: onProcessUsers,
  nextUser: onNextUser,
  complete: onComplete
};

var app = new App(eventHandlers); // create our test runner.

app.get('/tp/show/method', function (req, res) { // and invoke it.
  app.emit('get', {
    req: req,
    res: res
  });
  /* note:
       For this example, req and res are added to the evt
       but are ignored.

       In a working application, they would be used to
       return a result or an error, should the need arise,
       via res.send().
   */
});

Result

in app.get
in onGet
in onProcessUsers: { users: [ { _id: 1 }, { _id: 2 } ] }
in onNextUser: { users: [ { _id: 1 }, { _id: 2 } ], last_id: null, result: {} }
in onNextUser: { users: [ { _id: 2 } ],
    last_id: 1,
    result: { '1': { _id: 1 } } }
in onNextUser: { users: [],
    last_id: 2,
    result: { '1': { _id: 1 }, '2': { _id: 2 } } }
in onComplete: { users: [],
    last_id: 2,
    result: { '1': { _id: 1 }, '2': { _id: 2 } } }

Answer 3

Well, I get it. If the function AdrModel.find is async, you're setting this values always to the last user.

This occurs because a async function will be executed after the for block end. So, the value of the user_id in all AdrModel.find calls will be always the same, because the saved scope where the async call is executed. Let's say your users are this collection

[{_id: 0}, {_id:2}, {_id: 3}]

So the calls of AdrModel.find will always use user_id -> 3 value:

ret[users[user_id]._id]=adr; //this guy will use user_id == 3, three times

EDIT
To resolve your problem is simple, modularize your code. Create a function to do this resource gathering:

function setAdr(userId){
    AdrModel.find({ 'user_id': userId },function(err, adr) {
        ...
    }
}

And then, you call it in your 'for':

...
for(var user_id in users){
    setAdr(users[user_id].id_adr);
    ...

This way you'll save a safe scope for each async call.

Answer 4

This is a typical problem caused by trying to handle asynchronous code with synchronous means. Your entire attempt is unfixable, you need to scrap it.

One widely adopted way of handling asynchronous code without going insane is by using promises.

Here is what your code could look like if you used a promise library. For the sake of the example I'm using Bluebird here.

var findAddress = Promise.promisify(AdrModel.find);
var findUsers = Promise.promisify(User2Model.find);

// a helper function that accepts a user object and resolves the address ID
function attachAddrToUser(user) {
    return findAddress({
        user_id: user.id_adr
    }).then(function (address) {
        user.address = address;
        return user;
    }).catch(function (e) {
        console.error("error finding address for user ID " + user.id_user, e);
    });
}

findUsers().then(function (users) {
    var pending = [], id_user;
    for (id_user in users) {
        pending.push(attachAddrToUser(users[id_user]));
    }
    Promise.all(pending).then(function (users) {
        // all done, build and send response JSON here
    }).catch(function (e) {
        // don't forget to add error handling
    });
});

working jsFiddle over here: http://jsfiddle.net/Tomalak/2hdru6ma/

Note: attachAddrToUser() modifies the user object that you pass to it. This is not entirely clean, but it's effective in this context.