CODEWITHSUNDEEP
javascriptnode.jsexpressweb-scrapingpuppeteer
Netside
Netside

Reputation: 80

Is there anyway to make Puppeteer perform page actions (like expanding Wikipedia entries) before I make it take a screenshot of a page?

I tried Googling the answer to this, but only beginner tutorials seem to come up, and the advanced ones only talk about things like taking screenshots and a few other things.

I just want to have Puppeteer expand Wikipedia entries by clicking on them, before taking the screenshot I have it successfully programmed to take now. Is this possible?

Thanks!

Upvotes: 0

Views: 1197

Answers (1)

theDavidBarton
theDavidBarton

Reputation: 8851

Solution

By default the answer to the question would be: you can collect all collapsible heading elements to an array with a simple page.$$ (alias for document.querySelectorAll in the chrome api), then elementHandle.click on all of them in a for...of loop (array.map or forEach are not that suitable to do async clicks!) and finally create a screenshot with fullPage: true option.

Additional info about puppeteer bug and its workaround ⚠️

During testing I've found a puppeteer/chromium bug in the last step: fullPage: true, so it requires another approach to take the full screenshot of the page.

When [fullPage] true, [it] takes a screenshot of the full scrollable page [...] [source]

The full scrollable page's height is defined at navigation in puppeteer, not at the time we create the screenshot, so if we open more collapsed content after the first render: everything that would appear below the line of the original document.body.scrollHeight will be cut.

As a workaround we can recalculate scrollHeight (and scrollWidth) after all content is opened. I experienced some strange cropping if I didn't wait 1-2 seconds after the clicks, so I forced a page.waitFor(1500) before the recalculation. A page.setViewport should be applied on the device before the screenshot (page.screenshot should not have additional options like fullPage or clip together with the previous viewport setting!)

Example

const puppeteer = require('puppeteer')
const iPhone = puppeteer.devices['iPhone 6']

async function fn() {
  const browser = await puppeteer.launch({ headless: true })
  const page = await browser.newPage()

  await page.emulate(iPhone)
  await page.goto('https://en.m.wikipedia.org/wiki/Story_of_the_Year')
  await page.waitForSelector('.collapsible-heading')

  // collect all collapsed headings to an array and click all of them
  const headingHandles = await page.$$('.collapsible-heading')
  for (const el of headingHandles) {
    await el.click()
  }

  // let a bit time for scrollHeight to be finalized
  await page.waitFor(1500)

  // set viewPort to fullPage manually
  const scrollHeight = await page.evaluate(() => document.body.scrollHeight)
  const scrollWidth = await page.evaluate(() => document.body.scrollWidth)
  await page.setViewport({ width: scrollWidth, height: scrollHeight })

  await page.screenshot({ path: 'wiki.png' })
  await browser.close()
}
fn()

Upvotes: 2

Related Questions