CODEWITHSUNDEEP
web-scrapingaxioscorspostman
Dashiell Rose Bark-Huss
Dashiell Rose Bark-Huss

Reputation: 2965

Axios GET request resulting in 403. Postman works

I am trying get a page's HTML (on someone else's site) with an Axios GET request from my node server. But the request is returning a 403 error. The same request works on Postman.

 axios(
      'https://www.ssense.com/en-ca/women/product/laura-lombardi/gold-cable-chain-necklace/6378111',
      {
        headers: { // tried to fake the user-agent but this didn't change anything
          'User-Agent':
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36',
        },
      }
    )
      .then((result) => {
        console.log(result);
      })
      .catch((err) => {
        console.log(err);
      });

Some have suggested that this is a CORS issue. This confuses me. I thought cors was something that only effects browsers. If it is a CORS issue, why would it effect axios? Since I have no control over the site's server, I of course can't change their cors settings. So what is the solution?

Why can't I get the response in axios like I can in Postman? Is there some a way to get the HTML in node like postman does?

Upvotes: 0

Views: 2729

Answers (3)

Dashiell Rose Bark-Huss
Dashiell Rose Bark-Huss

Reputation: 2965

Adding mode: 'no-cors' to the headers made it intermittently work but I don't even know if that's a real header so I think it was just a coincidence (I think it's an axios option not a header). I then realized that Postman was also giving me 403.

Upon closer look the html: "Access to this page has been denied because we believe you are using automation tools to browse the website"

Says it can happen because:

  • javascript is disabled or blocked by an extension (ad blockers for example)
  • Your browser does not support cookies

Switching to puppeteer since puppeteer can load javascript, not sure axios can. But still had the issue. Going to look into this stackoverflow question How to avoid being detected as bot on Puppeteer and Phantomjs?

Upvotes: -1

rd05
rd05

Reputation: 70

Postman, as a developer tool and in contrast as browsers, does not enforce CORS. That is the reason why it works on Postman and why it does not work on a browser. I am not entirely sure about your context, but sometimes, the solution when we can not control the server or API we are requesting and getting CORS issues is using a proxy. Like this one, for example.

Upvotes: -1

Md. Fazlul Hoque
Md. Fazlul Hoque

Reputation: 16187

Because you have to send exact headers whatever server sends to the browser. I can solve this issue using scrapy . Thanks –

Upvotes: -1

Related Questions