CODEWITHSUNDEEP
node.jspuppeteer
iLinked
iLinked

Reputation: 151

How to get ALL request headers in Puppeteer

I'm trying to get ALL request headers to properly inspect the request, but it only returns headers like the User-Agent and Origin, while the original request contains a lot more headers.

Is there a way of actually getting all the headers?

For reference, here's the code:

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
    headless: false
});

const page = await browser.newPage();
page.on('request', req => {
   console.log(req.headers());
});
await page.goto('https://reddit.com');

Thanks in advance, iLinked

Upvotes: 12

Views: 17749

Answers (2)

user3064538
user3064538

Reputation:

You can switch from puppeteer to playwright, and then using Firefox (but not Chromium or WebKit) you'll get more headers:

import playwright from 'playwright';

(async () => {
    const browser = await playwright['firefox'].launch();
    const page = await browser.newPage();

    page.on('request', req => {
        console.log(req.headers());
    });
    await page.goto("https://example.com/");

    await browser.close();
})();

playwright['firefox'] output (on other sites I've seen cookies too):

{
  host: 'example.com',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0',
  accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
  'accept-language': 'en-US,en;q=0.5',
  'accept-encoding': 'gzip, deflate, br',
  connection: 'keep-alive',
  'upgrade-insecure-requests': '1'
}

vs. playwright['chromium'] output:

{
  'upgrade-insecure-requests': '1',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4421.0 Safari/537.36'
}

Upvotes: 1

Elia Weiss
Elia Weiss

Reputation: 9975

U can use the url https://headers.cloxy.net/request.php to see your headers

await page.goto('https://headers.cloxy.net/request.php');

U can also print to log

  console.log((await page.goto('https://example.org/')).request().headers());

Upvotes: 4

Related Questions