CODEWITHSUNDEEP
javascriptweb-scrapingevaluatepuppeteer
Cat Burston
Cat Burston

Reputation: 3053

How can I pass variable into an evaluate function?

I'm trying to pass a variable into a page.evaluate() function in Puppeteer, but when I use the following very simplified example, the variable evalVar is undefined.

I can't find any examples to build on, so I need help passing that variable into the page.evaluate() function so I can use it inside.

const puppeteer = require('puppeteer');

(async() => {

  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();

  const evalVar = 'WHUT??';

  try {

    await page.goto('https://www.google.com.au');
    await page.waitForSelector('#fbar');
    const links = await page.evaluate((evalVar) => {

      console.log('evalVar:', evalVar); // appears undefined

      const urls = [];
      hrefs = document.querySelectorAll('#fbar #fsl a');
      hrefs.forEach(function(el) {
        urls.push(el.href);
      });
      return urls;
    })
    console.log('links:', links);

  } catch (err) {

    console.log('ERR:', err.message);

  } finally {

    // browser.close();

  }

})();

Upvotes: 291

Views: 160866

Answers (7)

flozia
flozia

Reputation: 4638

You have to pass the variable as an argument to the pageFunction like this:

const links = await page.evaluate((evalVar) => {

  console.log(evalVar); // 2. should be defined now
  ...

}, evalVar); // 1. pass variable as an argument

You can pass in multiple variables by passing more arguments to page.evaluate():

await page.evaluate((a, b c) => { console.log(a, b, c) }, a, b, c)

The arguments must either be serializable as JSON or JSHandles of in-browser objects: https://pptr.dev/#?show=api-pageevaluatepagefunction-args

Upvotes: 456

Azriz
Azriz

Reputation: 3211

Slightly different version from @wolf answer above. Make code much more reusable between different context.

// util functions
export const pipe = (...fns) => initialVal => fns.reduce((acc, fn) => fn(acc), initialVal)
export const pluck = key => obj => obj[key] || null
export const map = fn => item => fn(item)
// these variables will be cast to string, look below at fn.toString()

const updatedAt = await page.evaluate(
  ([selector, util]) => {
    let { pipe, map, pluck } = util
    pipe = new Function(`return ${pipe}`)()
    map = new Function(`return ${map}`)()
    pluck = new Function(`return ${pluck}`)()

    return pipe(
      s => document.querySelector(s),
      pluck('textContent'),
      map(text => text.trim()),
      map(date => Date.parse(date)),
      map(timeStamp => Promise.resolve(timeStamp))
    )(selector)
  },
  [
    '#table-announcements tbody td:nth-child(2) .d-none',
    { pipe: pipe.toString(), map: map.toString(), pluck: pluck.toString() },
  ]
)

Also not that functions inside pipe cant used something like this

// incorrect, which is i don't know why
pipe(document.querySelector) 

// should be 
pipe(s => document.querySelector(s))

Upvotes: 0

Srinivas Reddy Thatiparthy
Srinivas Reddy Thatiparthy

Reputation: 11155

I have a typescript example that could help someone new in typescript.

const hyperlinks: string [] = await page.evaluate((url: string, regex: RegExp, querySelect: string) => {
.........
}, url, regex, querySelect);

Upvotes: 4

wolf
wolf

Reputation: 103

For pass a function, there are two ways you can do it.

// 1. Defined in evaluationContext
await page.evaluate(() => {
  window.yourFunc = function() {...};
});
const links = await page.evaluate(() => {
  const func = window.yourFunc;
  func();
});


// 2. Transform function to serializable(string). (Function can not be serialized)
const yourFunc = function() {...};
const obj = {
  func: yourFunc.toString()
};
const otherObj = {
  foo: 'bar'
};
const links = await page.evaluate((obj, aObj) => {
   const funStr = obj.func;
   const func = new Function(`return ${funStr}.apply(null, arguments)`)
   func();

   const foo = aObj.foo; // bar, for object
   window.foo = foo;
   debugger;
}, obj, otherObj);

You can add devtools: true to the launch options for test

Upvotes: 10

harrrrrrry
harrrrrrry

Reputation: 14467

It took me quite a while to figure out that console.log() in evaluate() can't show in node console.

Ref: https://github.com/GoogleChrome/puppeteer/issues/1944

everything that is run inside the page.evaluate function is done in the context of the browser page. The script is running in the browser not in node.js so if you log it will show in the browsers console which if you are running headless you will not see. You also can't set a node breakpoint inside the function.

Hope this can help.

Upvotes: 24

Grant Miller
Grant Miller

Reputation: 28999

Single Variable:

You can pass one variable to page.evaluate() using the following syntax:

await page.evaluate(example => { /* ... */ }, example);

Note: You do not need to enclose the variable in (), unless you are going to be passing multiple variables.

Multiple Variables:

You can pass multiple variables to page.evaluate() using the following syntax:

await page.evaluate((example_1, example_2) => { /* ... */ }, example_1, example_2);

Note: Enclosing your variables within {} is not necessary.

Upvotes: 61

Mehdi Raash
Mehdi Raash

Reputation: 8985

I encourage you to stick on this style, because it's more convenient and readable.

let name = 'jack';
let age  = 33;
let location = 'Berlin/Germany';

await page.evaluate(({name, age, location}) => {

    console.log(name);
    console.log(age);
    console.log(location);

},{name, age, location});

Upvotes: 95

Related Questions