CODEWITHSUNDEEP
node.js
testerson testing
testerson testing

Reputation: 49

Trying to crawl a website using puppeteer but getting a timeout error

I'm trying to search the Kwik Trip website for daily deals using nodeJs but I keep getting a timeout error when I try to crawl it. Not quite sure what could be happening. Does anyone know what may be going on?

Below is my code, I'm trying to wait for .agendaItemWrap to load before it brings back all of the HTML because it's a SPA.

function getQuickStar(req, res){
    (async () => {
        try {
          const browser = await puppeteer.launch();
          const page = await browser.newPage();
          const navigationPromise = page.waitForNavigation({waitUntil: "domcontentloaded"});
          await page.goto('https://www.kwiktrip.com/savings/daily-deals');
          await navigationPromise;
          await page.waitForSelector('.agendaItemWrap', { timeout: 30000 });

      const body = await page.evaluate(() => {
        return document.querySelector('body').innerHTML;
      }); 
      console.log(body);

      await browser.close();
    } catch (error) {
      console.log(error);
    }
  })();

}

Here's a link to the web page I'm trying to crawl https://www.kwiktrip.com/savings/daily-deals

Upvotes: 0

Views: 639

Answers (1)

Félix
Félix

Reputation: 154

It appear your desired selector is located into an iframe, and not into the page.mainframe. You then need to wait for your iframe, and perform the waitForSelector on this particular iframe.

Quick tip : you don't need any page.waitForNavigation with a page.goto, because you can set the waitUntil condition into the options. By default it waits for the page onLoad event.

Upvotes: 1

Related Questions