Figuring out Node.js + Redis + Express.js and lovely Asynchronous Callbacks

Question

First, forgive me for this is very unchartered territory for me...

Got the following:

app.get('/user/:user_id/followings', function(req, res) {
    var response = {}
      , userId = req.params.user_id
      , ids = req.param('id').split(',')
      , prefix = 'user'
      , suffix = 'follower_ids';

    for (var i=0; i<ids.length; i++) {
      var id = ids[i]
        , key = prefix+':'+ids[i];

      console.log('1: ' + key);
      checkMembership(userId, id, key, suffix, function(error, reply){
        response[key] = reply;
        console.log('2: ' + key + ': ' + reply);
      });
    }

    res.json(response);
  });

  function checkMembership(userId, id, key, suffix, callback) {
    var lookup = key+':'+suffix;

    client.sismember(lookup, userId, callback);
  }

For the following path: /user/1/followings?id=1,2,3,4,1000

I get this in the console:

1: user:1
1: user:2
1: user:3
1: user:4
1: user:1000
2: user:1000: 0
2: user:1000: 1
2: user:1000: 1
2: user:1000: 1
2: user:1000: 0

And this in the browser: {}

I get it's happening because stuff is occurring asynchronously... I'm just not sure how to fix it.

The keys logged in the checkMembership function are wrong. The console should output for the 2nd callback as follows:

2: user:1: 0
2: user:2: 1
2: user:3: 1
2: user:4: 1
2: user:1000: 0

Help!

Answer 1

You are right this happens because checkMembership is asynchronous. So your for loop is finished while checkMembership is still waiting for answers.

Lets start by simplifying your example:

for (i = 0; i < 3; i += 1) {     

  console.log('inside: ', i);  

  process.nextTick(function () {  
    console.log('outside: ', i);    
  });                           

};      

console.log('end');   

Output:

inside:  0
inside:  1
inside:  2  
end                              
outside:  2
outside:  2
outside:  2

So the first problem is that the callback is executed when the counter is already at 2. You can fix this by creating a closure.

for (i = 0; i < 3; i += 1) {

  console.log('inside: ', i);

  (function () {
    var j = i; 
    process.nextTick(function () {
      console.log('outside: ', j);
    });    
  })(); 

};    

console.log('end');                           

Output:

inside:  0 
inside:  1 
inside:  2  
end
outside:  0                         
outside:  1 
outside:  2 

This will fix your problem with response[key] = reply;.

The second problem with this example is that end gets logged before the async function is done. In your code this means that res.json will be called before checkMembership is done setting keys + replies.

So we need to find out when all callbacks are done before we can log end. Since you know how big your list is you also know how many callbacks there are. This means we can count each time a callback is done. If counts equal size you know all callbacks were called.

In code it looks like this:

var size = 3;                               
var counter = 0;                            

var globalCallback = function () {    

  counter += 1;                             

  if (counter === size) {                   
    console.log('end');                     
  }                                         

}                                           

for (i = 0; i < size; i += 1) {             

  console.log('inside: ', i);               

  (function () {                            
    var j = i;                              
    process.nextTick(function () {          
      console.log('outside: ', j);          
      globalCallback();                     
    });                                     
  })();                                     

};  

Our final output:

inside:  0
inside:  1
inside:  2
outside:  0
outside:  1
outside:  2
end

Instead of writing a counter like this each time you have a list of async functions I prefer to use a control flow library. Something like async.js is very popular.

What helped me was to have a more functional view on my code. You have to see it as a list of values and for each value you want to apply a function. That's also why async.js looks a bit like underscore.js.