Using yield to wait until async code is done

Question

I am trying to learn how to use generators and yield, so I tried the following but it doesn't seem to be working.

I am using the following function, which contains 2 async calls:

var client = require('mongodb').MongoClient;

$db = function*(collection, obj){
    var documents;
    yield client.connect('mongodb://localhost/test', function*(err, db){
        var c = db.collection(collection);
        yield c.find(obj).toArray(function(err, docs){
            documents = docs;
            db.close();
        });
    });
    return documents.length;
};

Then to make the call original call, I am doing this:

var qs = require("querystring");

var query = qs.parse("keywords[]=abc&keywords[]=123");
var total = $db("ads", {"details.keywords": {$in: query["keywords[]"]}});
console.log(total);

When I get my output back in the console, I get this:

{}

I was expecting a number such as 200. What is it that I am doing wrong?

Answer 1

var client = require('mongodb').MongoClient;

$db = function*(collection, obj){
    var documents;
    yield client.connect('mongodb://localhost/test', function*(err, db){
        var c = db.collection(collection);
        yield c.find(obj).toArray(function(err, docs){
            documents = docs;
            db.close();
        });
    });
    return documents.length;
};    
var qs = require("querystring");

var query = qs.parse("keywords[]=abc&keywords[]=123");
var total = $db("ads", {"details.keywords": {$in: query["keywords[]"]}});
console.log(total);

As is, total is the iterator for the $db generator function. You would retrieve its yield values via total.next().value. However, the mongodb library is callback based and as such, its functions do not return values, so yield will return null.

You mentioned you were using Promises elsewhere; I would suggest taking a look at the bluebird in particular its promisify functionality. Promisification inverts the callback model so that the arguments to the callback are now used to resolve the promisified function. Even better, promisifyAll will convert an entire callback based API.

Finally, bluebird provides coroutine functionality as well; however its coroutines must return promises. So, your code may be rewritten as follows:

var mongo = require('mongodb');
var Promise = require('bluebird');

//here we convert the mongodb callback based API to a promised based API
Promise.promisifyAll(mongo);

$db = Promise.coroutine(function*(collection, obj){
//existing functions are converted to promised based versions which have
//the same name with 'Async' appended to them
    return yield mongo.MongoClient.connectAsync('mongodb://localhost/test')
                .then(function(db){
                  return db.collectionAsync(collection);})
                .then(function(collection) {
                  return collection.countAsync();});
});

var qs = require("querystring");
var query = qs.parse("keywords[]=abc&keywords[]=123");
$db('ads',{"details.keywords": {$in: query["keywords[]"]}})
.then(console.log)

Answer 2

TL;DR

For the short answer, you're looking for a helper like co.

var co = require("co");
co(myGen( )).then(function (result) { });

---

But Why?

There is nothing inherently asynchronous about ES6 iterators, or the generators which define them.

function * allIntegers ( ) {
    var i = 1;
    while (true) {
      yield i;
      i += 1;
    }
}

var ints = allIntegers();
ints.next().value; // 1
ints.next().value; // 2
ints.next().value; // 3

The .next( ) method, though, actually lets you send data back in to the iterator.

function * exampleGen ( ) {
  var a = yield undefined;
  var b = yield a + 1;
  return b;
}

var exampleIter = exampleGen();
exampleIter.next().value; // undefined
exampleIter.next(12).value; // 13 (I passed 12 back in, which is assigned to a)
exampleIter.next("Hi").value; // "Hi" is assigned to b, and then returned

It might be confusing to think about, but when you yield it's like a return statement; the left hand side hasn't been assigned the value yet... ...and more importantly, if you had put the var y = (yield x) + 1; the parenthesis are resolved before the rest of the expression... ...so you return, and the +1 is put on hold, until a value comes back.
Then when it arrives (passed in, via the .next( )), the rest of the expression is evaluated (and then assigned to the left hand side).

The object that's returned from each call has two properties { value: ..., done: false }
value is what you've returned/yielded and done is whether or not it's hit the actual return statement at the end of the function (including implicit returns).

This is the part that can then be used to make this async magic happen.

function * asyncGen ( id ) {
  var key = yield getKeyPromise( id );
  var values = yield getValuesPromise( key );

  return values;
}

var asyncProcess = asyncGen( 123 );
var getKey = asyncProcess.next( ).value;

getKey.then(function (key) {
  return asyncProcess.next( key ).value;
}).then(function (values) {
  doStuff(values);
});

There's no magic.
Instead of returning a value, I'm returning a promise.
When the promise completes, I'm pushing the result back in, using .next( result ), which gets me another promise.

When that promise resolves, I push that back in, using .next( newResult ), et cetera, until I'm done.

---

Can we do better?

We know now that we're just waiting for promises to resolve, then calling .next on the iterator with the result.

Do we have to know, ahead of time what the iterator looks like, to know when we're done?

Not really.

function coroutine (iterator) {
  return new Promise(function (resolve, reject) {
    function turnIterator (value) {
      var result = iterator.next( value );
      if (result.done) {
        resolve(result.value);
      } else {
        result.value.then(turnIterator);
      }
    }

    turnIterator();
  };
}


coroutine( myGen ).then(function (result) { });

This isn't complete and perfect. co covers extra bases (making sure all yields get treated like promises, so you don't blow up by passing a non-promise value... ...or allowing arrays of promises to be yielded, which becomes one promise which will return the array of results for that yield ...or try/catch around the promise handling, to throw the error back into the iterator... yes, try/catch works perfectly with yield statements, done this way, thanks to a .throw(err) method on the iterator).

These things aren't hard to implement, but they make the example muddier than it needs to be.

This is exactly why co or some other "coroutine" or "spawn" method is perfect for this stuff.

The guys behind the Express server built KoaJS, using Co as a library, and Koa's middleware system just takes generators in its .use method and does the right thing.

---

But Wait, there's more!

As of ES7, it's very likely that the spec will add language for this exact use-case.

async function doAsyncProcess (id) {
  var key = await getKeyPromise(id);
  var values = await getValuesPromise(key);
  return values;
}

doAsyncProcess(123).then(values => doStuff(values));

The async and await keywords are used together, to achieve the same functionality as the coroutine-wrapped promise-yielding generator, without all of the external boilerplate (and with engine-level optimizations, eventually).

You can try this today, if you're using a transpiler like BabelJS.

I hope this helps.

Answer 3

Yield and generators have nothing to do with asynchrony, their primary purpose is to produce iterable sequences of values, just like this:

function * gen() {
  var i = 0;
  while (i < 10) {
    yield i++;
  }
}

for (var i of gen()) {
  console.log(i);
}

Just calling a function with a star (generator function) merely creates generator object (that is why you see {} in console), that can be interacted with using next function.

That said, you can use generator functions as an analogue of asynchronous functions, but you need a special runner, like co.