CODEWITHSUNDEEP
javascriptnode.jsweb-scrapingnightmare
Scraper321
Scraper321

Reputation: 23

WEB SCRAPING - nightmare js and request

I am using combination of nightmare, cheerio and request in NODEjs, for making custom web scraping bot... I did authentication and filter setup with nightmare js, and now I need to call function like 

    request(URL, function(err, response, body){
    if (err) console.error(err);

    var scraping = cheerio.load(body);
    .
    .
    .
    .

But problem is that I don't know how to forward loaded "body" (by nightmare). I can't use URL because it's dynamically generated content (tables), which means that URL is always the same... I tried to use this instead of URL, but it wont work. Any suggestions? Thank you

Upvotes: 0

Views: 2165

Answers (1)

Sachi
Sachi

Reputation: 1386

You don't need to use request. In fact, you shouldn't. Nightmare itself can pass the html data to cheerio.

Once you logged in and went to your desired webpage in nightmare, use evaluate to get the html. You can do something like this:

 nightmare
    .viewport(1280, 800)
    .goto(url)
    .wait('#emailselectorId')
    .type('#emailselectorId', 'theEmail\u000d')
    .type('#ap_password', 'thePassword\u000d')
    .click('#signInSubmit')

    //do something in the chain to go to your desired page.
    .evaluate(() => document.querySelector('body').outerHTML)
    .then(function (html) {
        cheerio.load(html);
        // do something 
    })
    .catch(function (error) {
    console.error('Error:', error);
    });

Upvotes: 3

Related Questions