CODEWITHSUNDEEP
javascriptjquerypuppeteer
user6122500
user6122500

Reputation: 942

Puppeteer: Cannot find element loaded by Javascript

I've been using Puppeteer to scrape some websites, and it works well when the element I need is in the DOM; however I can't get it to work when the element is loaded via Javascript. E.g. please see my code below. More specifically, the page.waitForSelector always triggers a timeout error. I've tried a page.screenshot and the resulting image does show a fully loaded page, which contains this .evTextFont element.

How can I modify this code to successfully retrieve the .evTextFont element?

I've tried both Puppeteer versions 1.11 and 1.17, but am getting the same problem for both

Thanks a lot

Adapted from here

const puppeteer = require('puppeteer');
const URL = 'https://www.paintbar.com.au/events-1/moments-in-moonlight';

puppeteer.launch({ headless: true, args: ['--no-sandbox', '--disable-setuid-sandbox'] }).then(async browser => {
    const page = await browser.newPage();
    await page.setViewport({width: 1200, height: 600})
    await page.goto(URL, {waitUntil: 'networkidle0'});
    await page.waitForSelector('.evTextFont');
    await page.addScriptTag({url: 'https://code.jquery.com/jquery-3.2.1.min.js'});
    // await page.screenshot({ path: './image.jpg', type: 'jpeg' });

    const result = await page.evaluate(() => {
        try {
            var data = [];

            $('.evTextFont').each(function() {
                const title = $(this).text();
                data.push({
                    'title' : title,
                });
            });
            return data;

        } catch(err) {
            console.log(err.toString());
        }
    });

    await browser.close();

    for(var i = 0; i < result.length; i++) {
        console.log('Data: ' + result[i].title);
    }

    process.exit();
}).catch(function(error) {
    console.error(error);
    process.exit();
});

Upvotes: 2

Views: 659

Answers (1)

Vaviloff
Vaviloff

Reputation: 16838

It happens because the event you're looking for is shown inside of an iframe element, from another site, so you need to find that iframe first and then do operation on it.

await page.goto(URL, {waitUntil: 'networkidle0'});

// Looking for the iframe with the event
const frame = (await page.frames()).find(f => f.url().includes("events.wix.com"));

// Then do work as before, but on that frame
await frame.waitForSelector('.evTextFont');
await frame.addScriptTag({url: 'https://code.jquery.com/jquery-3.2.1.min.js'});

const result = await frame.evaluate(() => {...})

enter image description here

Upvotes: 1

Related Questions