CODEWITHSUNDEEP
node.jschromiumpuppeteer
dinigo
dinigo

Reputation: 7448

Puppeteer fails to navigate to url (ERR_EMPTY_RESPONSE)

The problem This is the simplest code you can write to navigate to a page using puppeteer:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  page.on('response', response => {console.log(response.request().url())});
  page.on('error', error => {console.error(error.message)});
  await page.goto('https://www.vueling.com/es');
  await browser.close();
})();

But this generates an error:

UnhandledPromiseRejectionWarning: Error: net::ERR_EMPTY_RESPONSE at https://www.vueling.com/es

The question

This page loads perfectly when I try myself in Chrome/Chromium (~135 requests in 5 ~7"). My question is, is this a puppeteers thing, is Chrome's fault or is there something else I'm missing? Why does this not work?

Environment

Upvotes: 5

Views: 6230

Answers (2)

x-magix
x-magix

Reputation: 2860

Previous answer from @Romain 6 years ago want work today. Install npm user-agent and use it like this:

    const UserAgent = require("user-agents");

    const userAgent = new UserAgent({
      deviceCategory: "desktop",
      platform: "Linux x86_64",
    });

    await page.setUserAgent(userAgent.userAgent);

happy coding

Upvotes: 0

Romain
Romain

Reputation: 809

Some websites may be detecting puppeteer because it has a particular user-agent : Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/72.0.3617.0 Safari/537.36. You can notice the Headless Chrome in this one.

If you override your user-agent : await page.setUserAgent("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"); before loading the page with a regular browser user-agent, it works as expected.

If a website tries to detect puppeteer (there are plenty of other ways to do it), it is because they don't want their information to be accessed automatically. So if you run it on a website that you don't own, you should respect its data.

Upvotes: 2

Related Questions