CODEWITHSUNDEEP
javascriptnode.jses6-promisepuppeteercheerio
Kaleidics
Kaleidics

Reputation: 177

How to return a value using the Puppeteer library to scrape dynamic web pages?

I am using cheerio.js and puppeteer.js following this tutorial to try to do some basic web scraping. I'm not following exactly as the tutorial as I am trying to write it on the server side with the idea of having my backend handle all the scraping and then in the future pass that data to a front end.

As it is written now, I am getting

[nodemon] restarting due to changes... [nodemon] starting node server.js Your app is listening on port 8080 [Function]

Looks like the dynamicScraper is returning [Function] when I would expect it to be the html like in the tutorial?

Main server.js file

'use strict';

const express = require('express');
const cors = require('cors');
const app = express();
const cheerio = require('./potusScraper');

app.use(express.json());
app.use(
    cors({
        origin: ['http://localhost:3000']
    })
);

app.get('/', (req, res) => {
    let { scraper, dynamicScraper } = cheerio;
    //dynamicScraper should return html as a string?
    dynamicScraper()
        .then(html => {
            res.send(html);
        })
        .catch(err => {
            console.log(err);
        });
});

app.listen(process.env.PORT || 8080, () => {
    console.log(`Your app is listening on port ${process.env.PORT || 8080}`);
});

potusScraper.js file

'use strict';

const rp = require('request-promise');
const $ = require('cheerio');
const puppeteer = require('puppeteer');
const url = 'https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States';
const url2 = 'https://www.reddit.com';



const cheerio = {
    scraper: function() {
        return rp(url)
            .then(html => {
                const wikiUrls=[];
                for (let i = 0; i < 45; i++) {
                    wikiUrls.push($('big > a', html)[i].attribs.href);
                }
                return(wikiUrls);
            })
            .catch(err => console.log(err))
    },
    dynamicScraper: function() {
       return puppeteer //doesn't look like this works?
            .launch()
            .then(browser => {
                return browser.newPage();
            })
            .then(page => {
                return page.goto(url2)
                    .then(() => {return page.content});
            })
            .then(html => {
                console.log(html);
                return(html);
            })
            .catch(err => console.log(err));
    }

}

module.exports = cheerio;

Upvotes: 0

Views: 412

Answers (1)

Thomas Dondorf
Thomas Dondorf

Reputation: 25240

You are returning the page.content function in this code lines instead of calling it:

            .then(page => {
                return page.goto(url2)
                    .then(() => {return page.content});
            })

The third line should look like this:

                    .then(() => {return page.content()});

In addition, you could simplify your code by using a concise arrow function:

                    .then(() => page.content());

Upvotes: 1

Related Questions