CODEWITHSUNDEEP
javascriptnode.jsweb-scraping
JVG
JVG

Reputation: 21170

Building a simple Node.js scraper function

I'm tryign to build a very simple scraper function for nodeJS - just a function that I can pass a URL to, and it returns the scraped data as var data.

I'm completely new to Node.js and can't work out why the following isn't working:

var request = require('request');
var cheerio = require('cheerio');

function scrape(url) {
    console.log("Scraping: " + url);
    request(url, function(err, resp, body) {

            if (err) {
                throw err;
            }
            var html = cheerio.load(body);
            return html;
        });
}


var data = scrape('http://www.stackoverflow.com');

$ = data;
var logo = $('#hlogo a').text();
console.log(logo);

The above code should return "Stack Overflow" but obviously does not. When I run this in the console I get an error:

var logo = $('#hlogo a').text();
           ^
TypeError: Property '$' of object #<Object> is not a function

Any ideas why this isn't working for me?

Upvotes: 0

Views: 444

Answers (1)

Tommi
Tommi

Reputation: 3247

Your data will be undefined, because scrape function does not return a value, additionaly it asynchronous.

You need change logic to something like this:

function scrape(url, oncomplete) {
    console.log("Scraping: " + url);
    request(url, function(err, resp, body) {

        if (err) {
            throw err;
        }
        var html = cheerio.load(body);
        oncomplete(html);
    });
}


scrape('http://www.stackoverflow.com', function(data) { /* do work here*/ });

Upvotes: 2

Related Questions