CODEWITHSUNDEEP
javascriptnode.jsweb-scrapingpuppeteer
user2805620
user2805620

Reputation: 125

Puppeteer: Grabbing html from page that doesn't refresh after input tag button is clicked

I am trying to grab some html after a input tag button is clicked. I am clicking the button with page.evaluate() since page.click() does not seem to work for an input tag button. I have tried visual debugging with headless:false in the puppeteer launch options to verify that the browser indeed navigated to the point after the button is clicked. I am unsure as to why page.content() returns the html before the button is clicked rather than the html after the event happens.

const puppeteer = require('puppeteer');
const url = 'http://www.yvr.ca/en/passengers/flights/departing-flights';
const fs = require('fs');
const tomorrowSelector = '#flights-toggle-tomorrow'

puppeteer.launch().then(async browser => {
    const page = await browser.newPage();
    await page.goto(url);
    await page.evaluate((selector)=>document.querySelector(selector).click(),tomorrowSelector);
    let html = await page.content();

    await fs.writeFile('index.html', html, function(err){
        if (err) console.log(err);
        console.log("Successfully Written to File.");
    });
   await browser.close();
  });

Upvotes: 0

Views: 1537

Answers (1)

vsemozhebuty
vsemozhebuty

Reputation: 13792

You can click on the label for the radio. Also, you need to wait for some sign of changed state (for XHR/fetch response or new selectors). For example, this code works for me, but you can use any other condition or just wait for some seconds.

const fs = require('fs');
const puppeteer = require('puppeteer');

const url = 'http://www.yvr.ca/en/passengers/flights/departing-flights';

const tomorrowLabelSelector = 'label[for=flights-toggle-tomorrow]';
const tomorrowLabelSelectorChecked = '.yvr-form__toggle:checked + label[for=flights-toggle-tomorrow]';

puppeteer.launch({ headless: false }).then(async (browser) => {
  const page = await browser.newPage();
  await page.goto(url);
  await Promise.all([
    page.click(tomorrowLabelSelector),
    page.waitForSelector(tomorrowLabelSelectorChecked),
  ]);

  const html = await page.content();

  await fs.writeFile('index.html', html, (err) => {
    if (err) console.log(err);
    console.log('Successfully Written to File.');
  });
  // await browser.close();
});

Upvotes: 2

Related Questions