CODEWITHSUNDEEP
javascriptnode.jspuppeteer
almarc
almarc

Reputation: 1658

Puppeteer page.waitForNavigation() timeout error handling

Using puppeteer, i open up a page when i enter a value - it outputs the result.

await page.click('button[class="button form-button rs-gcbalance-btn"]')

await page.waitForSelector('div[class="small-4 large-4 rs-gcbalance-result-num-col').catch(err => console.log(err))

await page.evaluate((card) => {
    console.log(card + " - " + document.querySelectorAll('div[class="small-4 large-4 rs-gcbalance-result-num-col"]')[1].querySelector('span').innerHTML)
}, card)

But that works correctly only if the value on enter is valid. If it's not, it would throw an error, but without any network activity or load event.
That means, that if the value is incorrect, the element i'm waiting for won't appear and will throw an error, closing the program. 

Navigation Timeout Exceeded: 30000ms exceeded

The question is: how to handle the error, so if it throw a timeout error, i can catch it and call another function?

Upvotes: 20

Views: 27845

Answers (5)

Dhiraj Kumar
Dhiraj Kumar

Reputation: 31

you can use waitForNavigation method after button click and attach a catch block like below to handle navigation timeout error in case value entered is incorrect and navigation does not happens

page.waitForNavigation({waitUntil:"domcontentloaded"}).catch(error => {
    // handler code here
});

Upvotes: 3

Victor Cordeiro Costa
Victor Cordeiro Costa

Reputation: 2194

The try catch block is a solution.

However, on the accepted answer all errors are silenced!!

You should only catch the Puppeteer TimeOut errors.

try {
  await page.waitForSelector('.foo');
} catch (e) {
  if (e instanceof puppeteer.errors.TimeoutError) {
    // Do something if this is a timeout.
  }
}

References: https://devdocs.io/puppeteer/

Upvotes: 22

user1642018
user1642018

Reputation:

When i was trying to scrape some pages, i was facing similar issue as default timeout was 30000 ms i.e. 30 seconds and page was taking longer than 30 seconds to load, so basically there was 2 main issues.

  1. script was not getting killed after nodejs throwing page.waitForNavigation() timeout error, so it was keeping mysql connection active to the server and new connections was being created by cronjob and they were all in sleep condition.
  2. Page was not being scraped so needed to increase timeout.

here is my final code.

const puppeteer = require('puppeteer');
var mysql = require('mysql');


var mysql_con = mysql.createConnection({
  host: "",
  user: "",
  password: "",
  database: ""

});

//connect to mysql
mysql_con.connect(function(err) {
  if (err) throw err;
  console.log("Connected! to MySQL");
});


(async () => {

    const args = [
        '--no-sandbox', 
        '--disable-setuid-sandbox',
        '--disable-infobars',
        '--window-position=0,0',
        '--ignore-certifcate-errors',
        '--ignore-certifcate-errors-spki-list',
        '--ignoreHTTPSErrors=true',
        '--user-agent="Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z‡ Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"' 
    ];

    const browser = await puppeteer.launch({ args });
    const page = await browser.newPage();

    // Configure the navigation timeout to 2 minutes, becuase sometimes site is too busy
    await page.setDefaultNavigationTimeout(120000);

  try {

    // Now you can go wherever you want
    const response = await page.goto('https://www.example.com/');

    //print http status code 
    console.log(response.status());

    //do mysql related stuff here

    //close mysql connection
    mysql_con.end();

  } catch (e) {

    console.log('cant load the page, maybe server is busy : ' + e);

    //close mysql connection
    mysql_con.end();

    await browser.close();

    //double tap to die script
    process.exit();
  }



    // Then when you're done, just close
    await browser.close();
})();

here we fixed 2 issues.

  1. first issue by using try and catch block and killing script/ending mysql connection in catch block.

  2. increasing page timeout from 30 seconds default to 2 minutes.

    page.setDefaultNavigationTimeout(120000);

Upvotes: 1

SafiUllah
SafiUllah

Reputation: 9

Try uisng this will slove your problem. page.waitForNavigation( { timeout: 1000, waitUntil: 'domcontentloaded' });

Upvotes: -2

Everettss
Everettss

Reputation: 16059

Just wrap it in try catch block:

try {
  await page.waitForSelector('#element', { timeout: 1000 });
  // do what you have to do here
} catch (e) {
    console.log('element probably not exists');
}

Here is a fully working example:

const puppeteer = require('puppeteer');

const html = `
<html>
    <body>
        <div id="element">element inner html</div>
    </body>
</html>`;

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(`data:text/html,${html}`);

  try {
    await page.waitForSelector('#element-not-exists', { timeout: 1000 });
    const element = await page.$('#element-not-exists');
    console.log(await (await element.getProperty('innerHTML')).jsonValue());
  } catch (e) {
    console.log('element probably not exists');
  }
  await browser.close();
})();

Upvotes: 9

Related Questions