CODEWITHSUNDEEP
javascriptnode.jsrequestnpm
Rajit Garg
Rajit Garg

Reputation: 539

request node module not giving html

I am using request nodejs module to get html for a website as below:

var request = require('request');

request("http://www.thenewschool.org/", function (error, response, body) {
    if (!error && response.statusCode == 200) {
        console.log("body>>>>>>>>>>");
    } else {
        console.log("error>>>>>>>>>"+error);
        console.log("response statusCode>>>>>>>>>"+response.statusCode);
        console.log("response body>>>>>>>>>"+response.body);
    }
})

and this gives me this output

error>>>>>>>>>null

response statusCode>>>>>>>>>403

response body>>>>>>>>>Sorry, this request has been blocked due to an invalid user agent.

This is passing for most of the cases, but fails in this case, can someone help me to solve this.

Upvotes: 4

Views: 3854

Answers (2)

Rodrigo Medeiros
Rodrigo Medeiros

Reputation: 7862

You just have to pass the user-agent in the headers (because the URL requires it), like:

var options = {
  headers: {'user-agent': 'node.js'}
}

request("http://www.thenewschool.org/", options, function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log("body>>>>>>>>>>" + body);
  } else {
    console.log("error>>>>>>>>>"+error);
    console.log("response statusCode>>>>>>>>>"+response.statusCode);
    console.log("response body>>>>>>>>>"+response.body);
  }
})

Upvotes: 11

Jerome WAGNER
Jerome WAGNER

Reputation: 22442

well you are receiving an HTTP 403 error code : Access Is Forbidden.

this probably means that your request has been "profiled" as "we don't want you here" :

  • this can be because your IP has been flagged
  • or because your are missing a header that would make your request look like the one of a real browser. Most probably the user-agent header given the body of the answer

Upvotes: 5

Related Questions