Node.js: How do you handle callbacks in a loop?

Question

I'm using Node.js and Box SDK. My (failing!) code looks like this:

var connection = box.getConnection(req.user.login);
connection.ready(function () {
  connection.getFolderItems(0, null, function (err, result) {
    if (err) {
      opts.body = err;
    } else {
      opts.body = result;
      var a = [];
      for (var i=0; i < result.entries.length; i++) {
        connection.getFileInfo(result.entries[i].id, function (err, fileInfo) {
        if (err) {
          opts.body = err;
        } else {
          a.push(fileInfo);
        }
      });}
    }

In "procedural" terms, this is which I'm trying to do:

var connection= box.getConnection()
var items = connection.getFolderItems()
var itemList = new List
foreach (item in items) {
  connection.getFileInfo(item.id)
  itemList.add(item)
}
display(itemList)

My problem is that connection.getFolderItems() and connection.getFileInfo() are asynchronous - the "for" loop exits before all the results are returned.

Q: What is the best way in Node.js to 1) get the result of the first async call, 2) iterate through the list, making more async calls, and 3) process the results when everything is "done".

Q: Are promises a good choice here?

Q: Is done()/next() an option?

Q: Is there any "standard idiom" in Node.js for this kind of scenario?

Answer 1

Q: What is the best way in Node.js to 1) get the result of the first async call, 2) iterate through the list, making more async calls, and 3) process the results when everything is "done".

There are multiple approaches. Hand coding, Promises, Async library. "Best" is in the eye of the beholder so not really for us to say here. I use Promises for all my async coding. They have been officially standardized in ES6 and there are good, robust implementations (I like Bluebird for the extra features it has beyond the standard that simplify complex async tasks and for it's promisifyAll() feature which gives you a promise interface on any standard async operation that uses the async callback calling convention).

I'd advise against hand coding complicated async operations because robust error handling is very difficult and exceptions can be silently eaten inside of async callbacks leading to lost error handling and difficult debugging. The Async library is probably the best non-Promise way of doing things as it provides some infrastructure and synchronization features around async callbacks.

I personally prefer promises. I think we'll see more async APIs standardizing on returning a promise as time marches forward so I think it's a better choice for a forward-looking way to learn and program.

Q: Are promises a good choice here?

Yes, promises allow you to run a bunch of asynchronous operations and then use something like Promise.all() to know when they are all done. It will also collect all the results from all the async operations for you.

Q: Is done()/next() an option?

I'm not exactly sure what you are asking about here, but you can manually code async operations to either run in parallel and know when they are done or to run them sequentially and know when they are done. Promises do a lot more of this work for you, but you can do it manually without them.

Q: Is there any "standard idiom" in Node.js for this kind of scenario?

If using promises, there would be a common way to do this. If not using promises, there is probably not a "standard idiom" as there are many different ways to code it yourself.

Promise Implementation

Here's an example using the Bluebird Promise library in node.js:

var Promise = require('bluebird');
var connection = Promise.promisifyAll(box.getConnection(req.user.login));
connection.ready(function() {
    connection.getFolderItemsAsync(0, null).then(function(result) {
        return Promise.map(result.entries, function(item) {
            return connection.getFileInfoAsync(item.id);
        })
    }).then(function(results) {
        // array of results here
    }, function(err) {
        // error here
    });
});

Here's how this works:

1. Promisify the connection object so that all its methods have a version that returns a promise (just add "Async" onto the end of the method to call this promisified version).

2. Call getFolderItemsAsync() and its promise will resolve with the result.entries array

3. Run a map of that array, running all the operations in parallel and returning a promise that resolves with an array of ordered results when all the operations are done.

4. The actual result for each entry is achieved with connection.getFileInfoAsync().

5. Create a resolve handler and a reject handler. If any error occurs anywhere in the process, it will propagate up to the reject handler. If all operations are successful, the last resolve handler will be called with an ordered array of results.

The version above aborts if there's an error and you get no results other than the error code. If you want to continue with a null result if there's an error, then you can use something like this:

var Promise = require('bluebird');
var connection = Promise.promisifyAll(box.getConnection(req.user.login));
connection.ready(function() {
    connection.getFolderItemsAsync(0, null).then(function(result) {
        return Promise.map(result.entries, function(item) {
            return connection.getFileInfoAsync(item.id).catch(function(err){
                // post the results as null and continue with the other results
                return null;
            });
        })
    }).then(function(results) {
        // array of results here (some might be null if they had an error)
    }, function(err) {
        // error here
    });
});

---

Manually Coded Version

Here's a manually coded version. The key to this one is detecting when your async loop is done by comparing if (results.length === result.entries.length). Note: This has incomplete error handling which is one of the difficulties of coding this by hand and not using an async framework like promises.

var connection = box.getConnection(req.user.login);
connection.ready(function () {
    connection.getFolderItems(0, null, function (err, result) {
        if (err) {
            // do error handling here
            opts.body = err;
        } else {
            var results = [];
            for (var i = 0; i < result.entries.length; i++) {
                connection.getFileInfo(result.entries[i].id, function (err, fileInfo) {
                    if (err) {
                        // do error handling here
                        opts.body = err;
                        results.push(null);
                    } else {
                        results.push(fileInfo);
                    }
                    // if done with all requests
                    if (results.length === result.entries.length) {
                        // done with everything, results are in results
                        // process final results here
                    }
                });
            }
        }
    });
});

Answer 2

Promises are a great idea, but you may want to take a look at the async module, specifically the collection handlers. It allows you to run async calls against a list of 'things' and gives you a place to run a method when all of the async calls are done. Don't know if this is better than promises, but options are always nice.

// Should give you a rough idea
async.each(items, function (item, callback) {
  connection.getFileInfo(result, callback);
}, function (err) {
  console.log('All done');
});

https://github.com/caolan/async#each

Answer 3

Q: What is the best way in Node.js to 1) get the result of the first async call, 2) iterate through the list, making more async calls, and 3) process the results when everything is "done".

You could either use async library or promisify the function calls and use Promise instead. Both are easy to use.

Q: Are promises a good choice here?

Yes. But it requires you to promisify your method first before using.

Q: Is done()/next() an option?

From what I understand, it's an entirely different concept. The done here refers to a callback function which you can call after the method finishes. And next is usually used in express to pass on a request to the next route, which I think is not relevant to the question you're asking.

Q: Is there any "standard idiom" in Node.js for this kind of scenario?

It's usually referred to just "async" or "non blocking" calls