CODEWITHSUNDEEP
javascriptnode.jsgoogle-chromegoogle-chrome-devtoolspuppeteer
Gabriel Bigelow
Gabriel Bigelow

Reputation: 11

How do I get browser pages when using Puppeteer.connect()?

I'm working on some automation with Puppeteer and I need to create the browser object by using Puppeteer.connect({ browserWSendpoint: endpoint }), rather than Puppeteer.launch({ headless: false }).

The problem I'm having is that when I try to await browser.pages(), it seems like the promise never resolves and just hangs indefinitely, whereas if I were to create the browser object with Puppeteer.launch(), await browser.pages() returns basically immediately with an array of pages.

Below is the code that hangs, using Puppeteer.connect()

import puppeteer from 'puppeteer'

async function wsEndPoint() {
    // I have script running in the background that executes this terminal command
    // "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 "http://127.0.0.1:9222/json/version" "http://localhost:3000/"
    // to open a chrome instance that allows me to connect to it via Puppeteer
    // and opens another tab that holds my React App that I use to interact with
    // my puppeteer instance

    return await fetch('http://127.0.0.1:9222/json/version')
        .then((res) => res.json())
        .then((res) => res.webSocketDebuggerUrl)
}

async function myPuppeteerFunction() {
    console.log('program running')

    const browser = await puppeteer.connect({ browserWSEndpoint: await wsEndPoint() })
    console.log('browser created')
    const page = await browser.newPage()
    console.log('page created')
    console.log(await browser.pages())

    // Doesn't log because await browser.pages() seems to not resolve
    console.log('This should log if the program does not hang')
}

myPuppeteerFunction()

Meanwhile, using Puppeteer.launch() does not hang

import puppeteer from 'puppeteer'

async function myPuppeteerFunction() {
    console.log('program running')

    const browser = await puppeteer.launch({ headless: false })
    console.log('browser created')
    const page = await browser.newPage()
    console.log('page created')
    console.log(await browser.pages())

    // logs successfully
    console.log('This should log if the program does not hang')
}

myPuppeteerFunction()

I initially thought that my issue was that I'm using the 'puppeteer-extra' package and the 'puppeteer-extra-plugin-stealth', but the same issue happens with the official 'puppeteer' package, just like I included in the example code.

Looking at the docs, it shows that both the .launch and .connect methods return a promise that resolves to a Browser object, so I'm confused as to why one works and the other does not.

Upvotes: 0

Views: 912

Answers (1)

Gabriel Bigelow
Gabriel Bigelow

Reputation: 11

Edit: This solution MAY work for you. If it does not, you may be experiencing another similar issue that I ran into. Some captcha systems seem to make permanent changes to a browser instance that leaves it in a state that causes Puppeteer to hang whenever browser.pages(), browser.newPage() or Puppeteer.connect() are used. The only workaround that I have found so far is completely closing all browsers and restarting your remote debugging instance.

Solution provided by @ggorlen's comment:

I'm not sure what originally caused puppeteer to hang, but replacing the "browserWSEndpoint" launchOption puppeteer.connect({browserWSEndpoint: await wsEndPoint()})
with the "browserURL" launchOption
puppeteer.connect({browserURL: "http://127.0.0.1:9222/json/version" })
solved the hanging issue when awaiting the browser.pages() method.

I'm running Node v20.0.0 and this solution worked for me on Puppeteer ^22.12.0, as well as Puppeteer-extra ^3.3.6 with Puppeteer-extra-plugin-stealth ^2.11.2

import puppeteer from 'puppeteer'

async function myPuppeteerFunction () {
    console.log("program running")

    // code that caused puppeteer to hang
    // const browser = await puppeteer.connect({ browserWSEndpoint: await wsEndPoint() })
    const browser = await puppeteer.connect({ browserURL: "http://127.0.0.1:9222/json/version" })

    console.log("browser created")
    const page = await browser.newPage()
    console.log("page created")
    // does not hang anymore
    console.log(await browser.pages())

    // This will now log!
    console.log('This should log if the program does not hang')
}
myPuppeteerFunction()

EDIT: This code change did initially fix my issue, but I believe the actual issue was related to something on my own machine. Several hours after posting this solution, I attempted to recreate the issue so that I could open a ticket about the bug, but I was unable. Both my original codeblock and the solution codeblock work as expected, now. If you're experiencing this issue, try turning it off and on again. Close all your browsers, kill any node.js instances running your puppeteer program, and start from the beginning.

Upvotes: 0

Related Questions