CODEWITHSUNDEEP
javascriptnode.jsweb-scrapingpuppeteer
GVB
GVB

Reputation: 411

Detecting POST response arrival after clicking with Puppeteer

I'm trying to capture the information from a table but this table has a pagination button ("next , prev"). When I click the next button, no navigation occurs; it just creates a POST request that returns a new HTML table.

When I click the next button that causes a POST (not navigation), how can I wait to this POST to finish before trying to capture the data again from the next page?

Maybe I can detect some changes in the table element, but I don't know how.

What is the best approach for this problem?

Right now I'm doing this:

while (await page.$(NEXT_BUTTON_SELECTOR) !== null) {
  await page.click(NEXT_BUTTON_SELECTOR);
  await page.waitFor(2 * 1000);
  pageTableArray = getData();
}

but I'm not convinced that this is a good way to do it.

Upvotes: 3

Views: 1578

Answers (2)

Evan Zhang
Evan Zhang

Reputation: 11

You can use event requestfinished to capture data.

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', req => {
    console.log('request:', req.url())
    req.continue();
});
page.on('requestfinished', (req) => {
    console.log('finished:', req.url())
});
page.on('requestfailed', (req) => {
    console.log('failed:', req.url())
})
await page.goto(url);
await page.click(selector);

Upvotes: 1

browserless
browserless

Reputation: 2132

If await page.click(NEXT_BUTTON_SELECTOR); is causing a pageload to happen then whatever happens next in the script is lost. To get around that you'd have to do:

page.click(NEXT_BUTTON_SELECTOR); // Notice no `await`
await page.waitForNavigation();

Check out more about that API from the docs page here!

Upvotes: 1

Related Questions