CODEWITHSUNDEEP
javascriptnode.jsasync-await
matini250
matini250

Reputation: 25

NodeJS is it possible to handle multiple https requests and wait until all are resolved

I want to create a function that

It works when I check only one URL of the array. However, I cannot find a solution to go through all the elements of the array and wait until all responses have been received before returning the new array
Here is my code:

const https = require("https");

async function checkurl(url) {
  return new Promise((resolve, reject) => {
    let req = https.get(url, function (res) {
      if (res.statusCode < 400) {
        // check Statuscode
        console.log("RESPONSE! StatusCode: " + res.statusCode);
        filteredUrls.push(url);
        resolve(); // if response --> resolve promise
      }
    });
    req.on("error", function (e) {
      reject(e); // if error --> reject promise
    });
  }).catch(function (e) {
    console.log(e); //log promise error
  });
}

async function filterUrls(urlArray) {
  filteredUrls = [];

  await checkurl(urlArray[1]);

  return filteredUrls;
}

urls = [
  "https://google.com",
  "https://youtube.com",
  "https://microsoft.com",
  "https://filterUrlsle.com",
];

async function app() {
  console.log("begin");
  let result = await filterUrls(urls);
  console.log(result);
  console.log("continue");
}

app();

When I past only one element of the array I get this as output:

begin
RESPONSE! StatusCode: 301
[ 'https://youtube.com' ]
continue

However I would like to update my code to something like this:

async function filterUrls(urlArray) {
  filteredUrls = [];

  await urlArray.forEach(async function (element) {
    await checkurl(element);
  });

  return filteredUrls;
}

But it seems that the async await is somehow ignored, as I get the following output:

begin
[]
continue
URL does not exist
RESPONSE! StatusCode: 301
RESPONSE! StatusCode: 301
RESPONSE! StatusCode: 301

Is there a clean solution to this problem? Thanks in advance

Upvotes: 1

Views: 601

Answers (1)

T.J. Crowder
T.J. Crowder

Reputation: 1074485

forEach doesn't know anything about promises, so that code would just await undefined (since forEach returns undefined).

There are promise combinator functions. The one you're probably looking for, since you expect some of your processes to fail, is Promise.allSettled, which waits for all of the promises you pass into it to settle (whether fulfilled or rejected) and returns an array of results telling you whether they were fulfilled or rejected and what the fulfillment value or rejection reason is:

async function filterUrls(urlArray) {
    const results = await Promise.allSettled(urlArray.map(async (url) => {
        await checkUrl(url);
        return url;
    }));
    return results.filter(({status}) => status === "fulfilled")
                  .map(({value}) => value);
}

That code

  • Tries all the URLs in parallel
  • Waits until all of the tests are done
  • Filters out the ones that failed
  • Maps the {status: "fulfilled", value: "(url)"} objects from Promise.allSettled back to an array of just the URLs that passed

FWIW, the other combinators are:

Upvotes: 4

Related Questions