NodeJS: async loop that waits between iterations

Question

I'm trying to make some checks before saving an array of objects (objects[]) to the DB (mongoDB using mongoose):

• Those objects are already sorted by date, so objects[0].date is lower than objects[1].date.
• Each object should check that last related saved object has a different value (to avoid saving the same info two times). This means that I've to query to the DB before each save, to make that check, AND each of these object MUST be stored in order to be able to make the check with the right object. If objects are not stored in order, the last related saved object might not be the correct one.

In-depth explanation:

HTTP request is send to an API. It returns an array of objects (sortered by date) that I want to process and save on my Mongo DB (using mongoose). I've to iterate through all these objects and, for each:

• Look for the previous related object stored on DB (which COULD BE one of that array).
• Check some values between the 'already stored' and the object to save to evaluate if new object must be saved or could be discarded.
• Save it or discard it, and then jump to next iteration.

It's important to wait each iteration to finish because:

• Items on array MUST be stored in DB in order: first those which lower date, because each could be modified by some object stored later with a higher date.
• If next iteration starts before previous has finished, the query that searchs for the previous object could not find it if it hasn't been stored yet

Already tried:

Using promises or async/await on forEach/for loops only makes that iteration async, but it keeps launching all iterations at once.

I've tried using async/await functions inside forEach/for loops, even creating my own asyncForEach function as shown below, but none of this has worked:

Array.prototype.asyncForEach = function(fn) {
  return this.reduce(
   (promise, n) => promise.then(() => fn(n)),
  Promise.resolve()
 );
};

Test function:

let testArray = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];

testArray.asyncForEach(function(element) {
  setTimeout(() => {
    console.log(element);
  }, Math.random() * 500);
});

Provided example should show numbers on order in every case. It's not a problem if internal function (setTimeout in the example) should return a promise.

What I think I need is a loop that waits some function/promise between iterations, and only starts the next iteration when the first is already finished.

How could I do that? Thanks you in advance!

Answer 1

const myArray = ['a','b','c','d'];

async function wait(ms) { // comment 3
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function doSomething() {
  await myArray.reduce(async (promise, item) => {
    await promise; // comment 2
    await wait(1000);

    // here we could await something else that is async like DB call
    document.getElementById('results').append(`${item} `);
  }, Promise.resolve()); // comment 1
}


setTimeout(() => doSomething(), 1000);

<div id="results">Starting in 1 second <br/></div>

You can also use reduce and async await which you already said you've tried.

Basically, if you read how reduce works you can see that it accepts 2 parameters, first being callback to execute over each step and second optional initial value.

In the callback we have first argument being an accumulator which means that it accepts whatever the previous step returns or the optional initial value for first step.

1) You are giving initial value of promise resolve so that you start your first step.

2) Because of this await promise you will never go into next step until previous one has finished, since that is the accumulator value from previous step, which is promise since we said that callback is async. We are not resolving promise per say here, but as soon as the previous step is finish, we are going to implicitly resolve it and go to next step.

3) You can put for example await wait(30) to be sure that you are throttling the Ajax requests and not sending to many requests to 3rd party API's, since then there is no way that you will send more than 1000/30 requests per second, even if your code executes really fast on your machine.

Answer 2

In asyncForEach function you are resolving a Promise, setTimeout doesn't return a Promise.So if you convert your setTimeout to Promise. It will work as expected. Here is the modified code:

testArray.asyncForEach(function(element) {
 return new Promise((resolve, reject) => {
  setTimeout(() => {
   console.log(element);
   return resolve(element)
 }, Math.random() * 500);
})
});

Answer 3

Hm, ok i am not 100% sure if i understand your question in the right way. But if you try to perform an async array operation that awaits for your logic for each item, you can do it like follow:

async loadAllUsers() {
    const test = [1,2,3,4];
    const users = [];
    for (const index in test) {
      // make some magic or transform data or something else
      await users.push(test[index])
    }

    return users;
  }

Then you can simply invoke this function with "await". I hope that helps you.