NodeJS constructing array from asynchronious callbacks before returning

Question

I'm writing a function that's returning and array of values. Some of the values are calculated in a callback. But I don't know how to make the program asynchronious so all of my results are in the array, and not added after they're returned.

let array = [] 
for (stuff : stuffs) {
  if (condition) {
     array.add(stuff)
  } else {
     api.compute(stuff, callback(resp) {
          array.add(resp.stuff)
     }
  }
}
res.json({ "stuff": array })

In this example the array is written to the response before the async calls have finished.

How can I make this work asynchronously?

Answer 1

Here is an answer without package using callbacks

Create a function that's gonna recursively treat all your stuffs.

getArray(stuffs, callback, index = 0, array = []) {
  // Did we treat all stuffs?
  if (stuffs.length >= index) {
    return callback(array);
  }

  // Treat one stuff
  if (condition) {
    array.add(stuffs[index]);

    // Call next
    return getArray(stuffs, callback, index + 1, array);
  }

  // Get a stuff asynchronously
  return api.compute(stuffs[index], (resp) => {
    array.add(resp.stuff);

    // Call next
    return getArray(stuffs, callback, index + 1, array);
  });
}

---

How to call it?

getArray(stuffs, (array) => {
   // Here you have your array
   // ...
});

---

EDIT: more explanation

---

What we want to do to transform the loop you had into a loop that handle asynchronous function call.

The purpose is that one getArray call gonna treat one index of your stuffs array.

After treating one index, the function will call itself again to treat the next index, until all get treated.

-> Treat index 0 -> Treat index 1 -> Treat index 2 -> Return all result

We are using parameters to pass the infos through the process. Index to know which array part we have to treat, and array to keep a tract of what we did calculate.

---

EDIT: Improvement to 100% asynchronous soluce

---

What we have done here it's a simple transposition of your initial for loop into an asynchronous code. it can be improved so by making it totally asynchronous, which make it better but slightly more difficult.

For example :

// Where we store the results
const array = [];

const calculationIsDone = (array) => {
  // Here our calculation is done
  // ---
};

// Function that's gonna aggregate the results coming asynchronously
// When we did gather all results, we call a function
const gatherCalculResult = (newResult) => {
  array.push(newResult);

  if (array.length === stuffs.length) {
    callback(array);
  }
};

// Function that makes the calculation for one stuff
const makeCalculation = (oneStuff) => {
  if (condition) {
    return gatherCalculResult(oneStuff);
  }

  // Get a stuff asynchronously
  return api.compute(oneStuff, (resp) => {
    gatherCalculResult(resp.stuff);
  });
};

// We trigger all calculation
stuffs.forEach(x => x.makeCalculation(x));

Answer 2

You have to use one of the approaches:

• async library
• Promise.all
• coroutines/generators
• async/await

The most cool yet, I think, is async/await. First we modify your function, so it returns a promise:

const compute = function(stuff) {
  return new Promise( (resolve, reject) => {
    api.compute(stuff, callback(resp){
      resolve(resp.stuff)
    });
  });
};

Then we modify your route with async handler:

app.get('/', async function(req, res, next) {
  const array = [];
  for (const stuff of stuffs) {
    if (condition) {
      array.add(stuff);
    } else {
      const stuff = await compute(stuff);
      array.push(stuff);
    }
  }
  res.json({ stuff: array });
});

Note: You might need to update node version to latest.

UPDATE:

Those who are not awared, how event loop works, execute this snippet, and finish with that:

const sleep = async function(ms) {
  console.log(`Sleeping ${ms}ms`);
  return new Promise( resolve => setTimeout(resolve, ms));
};

async function job() {
  console.log('start');

  for (let t = 0; t < 10; t++) {
    await sleep(100);
  }
}

job();

console.log('oops did not expect that oO');

You will be surprised.