CODEWITHSUNDEEP
javascriptarraysnode.jses6-promise
Dantedt2
Dantedt2

Reputation: 33

Unknown length array used to execute promised functions sequentially

I'm making a function on a node.js server which reads a CSV file, I need to read all lines and execute several promised operations (MySQL queries) for each one (update or insert, then set an specified column which identifies this item as "modified in this execution") and once this finishes change another column on those not updated or inserted to identify this items as "deleted"

At first, the problem I had was that this CSV has millions of lines (literally) and a hundred of columns, so I run out of memory quite easily, and this number of lines can grow or decrease so I cannot know the amount of lines I will have to process every time I receive it.

I made a simple program that allows me to separate this CSV in some others with a readable amount of lines so my server can work with each one of them without dying, thus making an unknown amount of files each new file is processed, so now I have a different problem.

I want to read all of those CSVs, make those operations, and, once those operations are finished, execute the final one which will change those not updated/inserted. The only issue is that I need to read all of them and I cannot do this simultaneously, I have to make it sequentially, no matter how many they are (as said, after separating the main CSV, I may have 1 million lines divided into 3 files, or 2 millions into 6 files).

At first I though about using a forEach loop, but the problem is that, foreach doesn't respects the promisifying, so it will launch all of them, server will run out of memory when loading all those CSVs and then die. Honestly, using a while(boolean) on each iteration of the foreach to wait for the resolve of each promisified function seems pretty.... smelly for me, plus I feel like that solution will stop the server from working properly so I'm looking for a different solution.

Let me give you a quick explanation of what I want:

const arrayReader=(([arrayOfCSVs])=>{
initialFunction();
functions(arrayOfCSVs[0])
  .then((result)=>{
    functions(arrayOfCSVs[1])
      .then((result2)=>{
        functions(arrayOfCSVs[2])
        (OnAndOnAndOnAndOn...)
        .then((resultX)=>{
          executeFinalFunction();
        });
    });
});

Upvotes: 0

Views: 57

Answers (1)

AvcS
AvcS

Reputation: 2323

You can use Array.reduce to get the previous promise and queue new promise, without the need for waiting.

const arrayReader = ([arrayOfCSVs]) => {
  initialFunction();
  return arrayOfCSVs.reduce((prom, csv) => {
    return prom.then(() => functions(csv));
  }, Promise.resolve()).then(resultX => executeFinalFunction());
}

Upvotes: 1

Related Questions