CODEWITHSUNDEEP
javascriptnode.jstypescriptxpathpuppeteer
Fredrik Enetorp
Fredrik Enetorp

Reputation: 425

Pass a function to evaluate() in puppeteer

I'm working on a web scraper in Typescript using Puppeteer and made a helper function like this: 

function evaluateXPath(page: puppeteer.Page, xpath: string): Promise<any[]> {
    return page.waitForXPath(xpath)
        .then(() => page.$x(xpath))
        .then(handles => page.evaluate((...handles) => handles.map(h => h.href), ...handles))
}

It works fine if I want to get the value of the href-field of an xpath, but I want to make it more generic. Something like this (replaced 'h => h.href' inside map() with 'evalFunc'): 

function evaluateXPath(page: puppeteer.Page, xpath: string, evalFunc: (value: any, index: number, array: any[]) => any): Promise<any[]> {
    return page.waitForXPath(xpath)
        .then(() => page.$x(xpath))
        .then(handles => page.evaluate((...handles) => handles.map(evalFunc), ...handles))
}

This obviously doesn't work since 'evalFunc' doesn't exist in browser context, but any attempts at passing the function to the browser fails or doesn't compile. Passing it inside the evaluate() function doesn't work since it's not assignable to a parameter of type 'SerializableOrJSHandle' and I've had no luck with the exposeFunction() function either.

Can someone show me how it's done?

Upvotes: 0

Views: 1610

Answers (1)

vsemozhebuty
vsemozhebuty

Reputation: 13782

I can think of two dubious and inconvenient solutions.

  1. Using eval() or Function() constructor inside the evaluated function and transfer the function code as a string (difficult for code writing, linting etc. — this can be mitigated a bit by using Function.prototype.toString()).

  2. Placing all the utility functions in a file and using page.addScriptTag() to inject this file so that the evaluated function can call any of the functions declared in this file (this can pollute the global scope).

Edit: 

function evaluateXPath(page: puppeteer.Page, xpath: string, evalFunc: (value: any, index: number, array: any[]) => any): Promise<any[]> {
    return page.waitForXPath(xpath)
        .then(() => page.$x(xpath))
        .then(handles => page.evaluate((evalFuncStr, ...handles) =>
            handles.map(eval(`(${evalFuncStr})`)),
            evalFunc.toString(),
            ...handles))
}

Upvotes: 3

Related Questions