CODEWITHSUNDEEP
javascriptnode.jspuppeteer
MaxioN
MaxioN

Reputation: 31

Node js Puppeteer goto array of pages

I try to go page by page from my array, but get this:

(node:4196) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 request listeners added. Use emitter.setMaxListeners() to increase limit (node:4196) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 framedetached listeners adde d. Use emitter.setMaxListeners() to increase limit (node:4196) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 lifecycleevent listeners add ed. Use emitter.setMaxListeners() to increase limit (node:4196) UnhandledPromiseRejectionWarning: Error: Protocol error (Page.navigate): Target closed. at Promise (D:\Kutz\irrParse\node_modules\puppeteer\lib\Connection.js:198:56) at new Promise () at CDPSession.send (D:\Kutz\irrParse\node_modules\puppeteer\lib\Connection.js:197:12) at navigate (D:\Kutz\irrParse\node_modules\puppeteer\lib\Page.js:520:39) at Page.goto (D:\Kutz\irrParse\node_modules\puppeteer\lib\Page.js:500:7) at uniqueLinks.forEach (D:\Kutz\irrParse\scrape.js:26:16) at Array.forEach () at D:\Kutz\irrParse\scrape.js:25:15 at at process._tickCallback (internal/process/next_tick.js:118:7) (node:4196) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (r ejection id: 1) (node:4196) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise reject ions that are not handled will terminate the Node.js process with a non-zero exit code. (node:4196) UnhandledPromiseRejectionWarning: Error: Navigation Timeout Exceeded: 30000ms exceeded at Promise.then (D:\Kutz\irrParse\node_modules\puppeteer\lib\NavigatorWatcher.js:71:21) at 

const puppeteer = require("puppeteer");
var forEach = require('async-foreach').forEach;


const url = "https://reddit.com/r/programming";
const linkSelector = ".content a.title";

(async () => {
  // Launch chrome process
  const browser = await puppeteer.launch({headless: true});
  const page = await browser.newPage();

  await page.goto(url, { waitUntil: "load" });

  // This runs the `document.querySelectorAll` within the page and passes
  // the result to function
  const links = await page.$$eval(linkSelector, links => {
    return links.map((link) => link.href);
  });

  // Make sure we get the unique set of links only
  const uniqueLinks = [...links];
  //console.log(uniqueLinks[0]);

  uniqueLinks.forEach(async (link) => {
    await page.goto(link, { waitUntil: "load" });
  });

  // Kill the browser process
  await browser.close();
})();

Error throws in forEach()

Upvotes: 1

Views: 3327

Answers (1)

Chad Callahan
Chad Callahan

Reputation: 106

Unfortunately, Array.prototype.forEach's iterator function is not executed in an async manner as you would expect when defining it as async. Using a for loop should work for what you're trying to do.

for (let i = 0; i < uniqueLinks.length; i ++) {
  await page.goto(uniqueLinks[i], { waitUntil: "load" });
}

Upvotes: 9

Related Questions