CODEWITHSUNDEEP
javascriptnode.jssleep
Curtis
Curtis

Reputation: 2704

Waiting for data to return in NodeJS instead of Sleeping

So I'm writing up a little scraping tool that requires me to confirm an email address, however I'm using an API that doesn't update for maybe a few seconds after receiving the email.

The current method I'm using is this:

//wait for page 3 to finish loading
await Promise.all([
    page.waitForNavigation({ waitUntil: 'load' }),
    page.click('#submitbutton'),
]);

//sleep so we can make sure we receive the email.
await Apify.utils.sleep(5000);

//get emails
try {
    emails = await getEmails(userProfile.email); //this is just an Axios request/response.
} catch (error) {
    return res.send(error_response('email_api_failed'));
}

emails.data.forEach(obj => {
    //perform magic here on emails..
});

However I'll often get an error emails.data.forEach is not a function so what would be the correct approach?

Upvotes: 0

Views: 108

Answers (1)

Eyal Cohen
Eyal Cohen

Reputation: 1288

You might want to implement a retry functionality after sleeping.

If you didn't receive any response (where has data is undefined), try and request again in intervals of X milliseconds.

It can be easy as:

function getEmails(retries = 5) {
  return new Promise(async (resolve, reject) => {
    while (retries-- > 0) {
      await Apify.utils.sleep(5000)
      emails = await getEmails(userProfile.email)

      if (emails.data) {
        return resolve(emails.data)
      }
    }

    resolve('No Data')
  })
}

const data = await getEmails()

Upvotes: 2

Related Questions