CODEWITHSUNDEEP
javascriptnode.jsaxios
junvar
junvar

Reputation: 11574

Node `https.get` returns 403, `axios.get` and `curl` return 200

I'm trying to perform a GET request using Node's native https module and a 3rd party axios module (which is pretty commonly used):

const endpoint = 'https://www.pathofexile.com/api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403

The axios request works fine as expected; I can view response.data as well. But the https request fails with a 403.

Since https used to work, my hunch is that the server must have changed something, and that https.get and axios.get must be doing something slightly differently (e.g. sending different headers by default) that the server is no longer indifferent to.

Upvotes: 5

Views: 2568

Answers (2)

Marco Bonelli
Marco Bonelli

Reputation: 69286

After a little bit of testing, I noticed that by default the https module does not include an User-Agent header, while axios and curl both do. The server is rejecting your request because you have no User-Agent set.

To solve this, simply specify a custom User-Agent of your choice:

require('https')
    .get(endpoint, {headers: {'User-Agent': 'whatever'}},
        response => console.log('https', response.statusCode));

Upvotes: 9

junvar
junvar

Reputation: 11574

After few web results focusing on node https didn't find an answer, I found another stackoverflow post about python gets returning 403 while curl returns 200.

It turns out it was an issue with the request's 'User-Agent' header.

Curl by default sends the user agent 'curl/7.72.0':

> GET /94548/2401383/manifest.js HTTP/1.1
> Host: cdn.flashtalking.com
> User-Agent: curl/7.72.0
> Accept: */*

Axios by default sends the user agent 'axios/0.20.0\r\n':

_header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'User-Agent: axios/0.20.0\r\n' +
      'Host: www.pathofexile.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',

While node's https by default sends no user agent:

  _header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
    'Host: www.pathofexile.com\r\n' +
    'Connection: close\r\n' +
    '\r\n',

Apparently, the server now checks for a non empty User-Aagent header. Updating the https request to do so gets a 200:

const endpoint = 'https://www.pathofexile.com/api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403


require('https')
    .get(endpoint, {header: {headers: {'User-Agent': '_'}}}},
        response => console.log('https', response.statusCode));
// prints 200

Upvotes: 3

Related Questions