CODEWITHSUNDEEP
node.jspuppeteer
torbenrudgaard
torbenrudgaard

Reputation: 2561

Puppeteers waitFor functions fail BEFORE the page finished rendering

How come waitForFunction, waitForSelector, await page.evaluate etc. all give errors UNLESS I put a 10 seconds delay after reading the page?

I would think these were made to wait for something to happen on the page, but without my 10 seconds delay (just after page.goto) - all of them fail with errors. 

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    await page.goto('https://sunnythailand.com')

    console.log("Waiting 10 seconds")
    await new Promise((resolve)=>setTimeout(()=> resolve() ,10000));
    console.log("Here we go....")

    console.log("waitForFunction START")
    await page.waitForFunction('document.querySelector(".scrapebot_description").textContent.length > 0');
    console.log("waitForFunction FOUND scrapebot")

    console.log("Waiting for evaluate")
    const name = await page.evaluate(() => document.querySelector('.scrapebot_description').textContent)
    console.log("Evaluate: " + name)

    await browser.close()
})()

My theory is that our sunnythailand.com page sends an "end of page" or something BEFORE it finished rendering, and then all the waitFor functions go crazy and fail with all kinds of strange errors.

So I guess my question is... how do we get waitFor to actually WAIT for the event to happen or class to appear etc...?

Upvotes: 1

Views: 440

Answers (1)

satendra
satendra

Reputation: 113

Don't use time out cause you don't know how much time for it will take to load full page. It depends person to person on his internet bandwidth.

All you need to rely on promise for your class

await page.waitForSelector('.scrapebot_description');

Let's wait for your particular class then it will work fine.

Please remove this

//await new Promise((resolve)=>setTimeout(()=> resolve() ,5000));

Please let me know your test result after this. I am sure it will solve.

Upvotes: 2

Related Questions