CODEWITHSUNDEEP
node.jsweb-scrapingpuppeteercheerio
m1i2t3
m1i2t3

Reputation: 45

Puppeteer webscraping missing part of the html

I am trying to web scrape this website: https://kineticlabs.store/switches. However, when I run the code below, the html is missing many child div elements compared to inspecting the webpage on Chrome. I am very confused as to why it is because it only seems to be happening on this specific website. Thanks in advance.

let page = await configureBrowser(url);
    await page.reload();
    let html = await page.evaluate(() => document.body.innerHTML);

Upvotes: 0

Views: 383

Answers (1)

Vaviloff
Vaviloff

Reputation: 16838

The linked site is a single-page application all of which is rendered with javascript, so you need to give it time to load, parse and render.

It is best done waiting for an element that is guaranteed to be there only after the application is started.

await page.reload();

// Wait until #root is populated by the app
await page.waitForSelector('#root > div');

const products = await page.evaluate(() =>
  [...document.querySelectorAll('.hwsyok')].map(
    (product) => product.innerText
  )
);

Upvotes: 2

Related Questions