CODEWITHSUNDEEP
javascriptnode.jsdomweb-scrapingpuppeteer
Devonnnn
Devonnnn

Reputation: 25

Puppeteer always undefined but devtools works for nested node lists

I am new to Puppeteer and am not quite sure what I'm doing wrong here, but in the DevTools, I get the correct output. However, I am looking to create a file with the values and keep getting undefined.

I think it has something to do with node lists and not being able to return them, but I have no idea how to fix it.

This works in the the DevTools:

let arr2 = Array.from(document.querySelectorAll(
  "#data > div.data-wrapper > div > div > table > tbody  tr"))
.map(row => (
  {site:row.querySelector('td:nth- child(2)').innerText,
  pass:row.querySelector('td:nth- child(10)').innerText,
  user:row.querySelector('td:nth-child(9)').innerText
  }))
  //with a console.log()

I also have tried row.evaluate(()=>) which was not working for me.

Here is my code that is not working:

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
    await page.authenticate({ username: "username", password: "password" });
    await page.goto("https://website/.../all");
    await page.waitFor(120000); // beacuse It loads everything slowly and times out the default 30000

    console.log("started evalating");
    var data = await page.evaluate(() => {
      Array.from(
        document.querySelectorAll(
          "#data > div.data-wrapper > div > div > table > tbody  tr"
        )
      ).map(row => {
        return {
          site: row.querySelector("td:nth-child(2)").innerText,
          pass: row.querySelector("td:nth-child(10)").innerText,
          user: row.querySelector("td:nth-child(9)").innerText
        };
      });
    });
  console.log(data);
})();

//I want an array of objects but the result throws errors or comes back with [undefined,......]

Upvotes: 0

Views: 1211

Answers (1)

Grant Miller
Grant Miller

Reputation: 29037

The page function passed to page.evaluate() is missing a return statement, and therefore does not return a value. As a result, the variable data is undefined.

There are two additional details that you should note:

  1. Array.from() has a built-in map function.

    Array.from(arrayLike, mapFn)     // good
Array.from(arrayLike).map(mapFn) // bad

  2. You should use let or const instead of var whenever possible.

    let data = ... // good
var data = ... // bad

Here is a revision of your code that should work properly:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
  });

  const page = await browser.newPage();

  await page.authenticate({
    username: 'username',
    password: 'password',
  });

  await page.goto('https://website/.../all');

  await page.waitFor(120000);

  console.log('started evalating');

  let data = await page.evaluate(() => {
    return Array.from(
      document.querySelectorAll('#data > div.data-wrapper > div > div > table > tbody tr'),
      row => ({
        site: row.querySelector('td:nth-child(2)').innerText,
        pass: row.querySelector('td:nth-child(10)').innerText,
        user: row.querySelector('td:nth-child(9)').innerText,
      })
    );
  });

  console.log(data);
})();

Upvotes: 3

Related Questions