CODEWITHSUNDEEP
jquerynode.jsweb-scrapingpuppeteer
SIM
SIM

Reputation: 22440

Script unable to yield the desired result

I've written a script in node.js in association with puppeteer to scrape the first title out of several posts from a webpage. When I execute my following script, It neither fetches any result nor throws any error.

This is my try:

const puppeteer = require('puppeteer');
const url = "https://stackoverflow.com/questions/tagged/web-scraping";

(async function main() {
    const browser = await puppeteer.launch();
    const page    = await browser.newPage();
    page.on('console', obj => console.log(obj._text));
    await page.goto(url);
    await page.waitForSelector('.question-hyperlink');
    await page.$(() => {
        let item = $eval('.question-hyperlink').innerText;
        console.log(item);
    })
    await browser.close();
})();

Although I know that if I bring about the following change into my above script, it will work. However, I would like to stick to the way I tried above so that I can understand how .$eval() works in this very case.

await page.evaluate(() => {
    let item = document.querySelector('.question-hyperlink').innerText;
    console.log(item);
})

Upvotes: 1

Views: 88

Answers (1)

Vaviloff
Vaviloff

Reputation: 16838

page.$(selector) means that you want to get the handle of the first element with the specified selector, but you haven't given it any selector, that's why it won't work.

Instead you can get the handle of the needed element:

const link = await page.$('.question-hyperlink');

Then obtain a handle of the needed property of that element, and finally its value:

let valueHandle = await link.getProperty('textContent');
console.log(await valueHandle.jsonValue());

A simpler, cleaner solution

You can give page.$eval the desired element selector and a function to run on the element when it is found.

const linkTitle = await page.$eval('.question-hyperlink', el => el.textContent);
console.log(linkTitle);

Upvotes: 1

Related Questions