CODEWITHSUNDEEP
node.jsweb-crawlerpuppeteer
kietheros
kietheros

Reputation: 463

Puppeteer page.setRequestInterception didn't catch any requests on iframe

I'm crawling data from a page which contains an iframe. The sample code

await page.setRequestInterception(true);
page.on('request', request => {
    console.log(request.url());
    //.........................
    request.continue();
});

await page.goto(url, {waitUtil: 'networkidle0', timeout: 30000});
//..................

This code only showed requests on main frame, it couldn't catch any requests from the inner iframe. In Chrome network inspection, I saw many XHR requests from this page.

Is there a way to catch requests on iframe ? Many thanks !

Upvotes: 2

Views: 4551

Answers (2)

Ilya Shevyryaev
Ilya Shevyryaev

Reputation: 822

Seems that adding these to browser startup options may fix the problem:

args: [
   '--disable-web-security',
   '--disable-features=IsolateOrigins,site-per-process'
]

Update 2023: Seems Puppeteer will remove this in future versions. At the moment you can enable intercepting by adding this extra argument:

--disable-site-isolation-trials

Upvotes: 2

kietheros
kietheros

Reputation: 463

When I set puppeteer headless = true, the above code has worked. It catches all requests on all iframes on my crawling website.

Upvotes: 5

Related Questions