CODEWITHSUNDEEP
node.jsweb-scrapingpuppeteer
MITHU
MITHU

Reputation: 164

Can't grab a name from a webpage using puppeteer

I've created a script in node in combination with puppeteer to fetch a name which is populated upon filling in an input in a webpage. Here is how you can get that name - after opening that webpage, put 16803 right next to CP Number and hit the search button.

My following script can successfully fill in the input and populate the name by pressing the search button. What I can't do is fetch the name of the result. The selector I've defined is flawless. I would like to fetch the name on the fly without using return.

I've attempted so far:

const puppeteer = require("puppeteer");
const url = "https://www.icsi.in/student/Members/MemberSearch.aspx";

(async () => {
    const browser = await puppeteer.launch({headless:false});
    const [page] = await browser.pages();
    await page.goto(url, {waitUntil: 'networkidle2'});
    await page.type('input[name="dnn$ctr410$MemberSearch$txtCpNumber"]', '16803');
    await page.click('a#dnn_ctr410_MemberSearch_btnSearch');
    await page.waitFor('.name_head > span');
    await page.evaluate(async () => {
        const elem = document.querySelector('.name_head > span').innerText;
        console.log(elem);
    });
    await page.close();
})();

The above script gives me a blank output.

How can I fetch the name from that site?

Upvotes: 1

Views: 249

Answers (1)

Md. Abu Taher
Md. Abu Taher

Reputation: 18866

.evaluate will run the script inside the browser. You need to do one of below,

Return and log it outside

Most of the time this is your intended option. You will want to get the name and use it later on. You can return the value or a JSHandle.

const elemText = await page.evaluate(async () => document.querySelector('.name_head > span').innerText);
console.log(elemText);

Enable console with dumpio

When you are launching the browser, use dumpio: true as option. It will pipe the browser process stdout and stderr into process.stdout and process.stderr. Defaults to false.

puppeteer.launch({dumpio:true})

Enable console with page.on('console')

page.on('console', msg => console.log('PAGE LOG:', msg.text()));

Watch the browser console yourself

Launch the browser with headless: false and devtools: true, you can see the logs yourself.

puppeteer.launch({ headless: false, devtools: true })

Upvotes: 3

Related Questions