CODEWITHSUNDEEP
javascriptnode.jstimeoutsettimeoutpuppeteer
Daniel Rika
Daniel Rika

Reputation: 69

How to stop NodeJS script from crashing when timeout exceeded

Below you can find the error that occurs.

I'm trying to scrape contents of a website using NodeJS and puppeteer. Sometimes the code stops with the error Timeout Exceeded. Is there a way so that if the timeout of the page load is exceeded I can run a function that would either reload the page or have the script wait for a few seconds and then reload the page until it gets the data correctly, without crashing? If so, how would I go forward with implementing it?

Thank you.

(node:8300) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (C:\Users\danie\node_modules\puppeteer\lib\LifecycleWatcher.js:143:21)
  -- ASYNC --
    at Frame.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:108:27)
    at Page.goto (C:\Users\danie\node_modules\puppeteer\lib\Page.js:656:49)
    at Page.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:109:23)
    at scrape (C:\Users\danie\Documents\Node Projects\p-download.js:23:14)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:8300) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8300) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

My Code:

const puppeteer = require('puppeteer');

let scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    await page.setRequestInterception(true);    
    page.on('request', (req) => {
        if(req.resourceType() == 'stylesheet' || req.resourceType() == 'script' || req.resourceType() == 'font' || req.resourceType() == 'media' || req.resourceType() == 'image'){
            req.abort();
        }
        else {
            req.continue();
        }
    }); //Disables loading CSS, images and scripts

    for(i=0; i<5000; i++){
        await page.goto('https://website.com/' + i);
        let result = await page.evaluate(() => {
            var result = '';
            for (i=1; i<=10; i++){
                result += document.getElementsByTagName('td')[i].innerText;
                result += ',';
            }
            result += '\n';
            return result;
        });
    }
}
scrape();

Upvotes: 1

Views: 2006

Answers (1)

hretic
hretic

Reputation: 1065

put your code in try/catch block to avoid the crash ... i would move in loop code into another function 

for(i=0; i<5000; i++){
    result = await open_page(page , i );
}


async function open_page(page , i ){
    try {
            await page.goto('https://website.com/' + i);
            let result = await page.evaluate(() => {
                var result = '';
                for (i=1; i<=10; i++){
                    result += document.getElementsByTagName('td')[i].innerText;
                    result += ',';
                }
                result += '\n';
                return result;
            });


            return {stat:1 , result : result } ;

    }
    catch(e){
        return {stat:0 , error : e } ;
    }

}

Upvotes: 3

Related Questions