CODEWITHSUNDEEP
node.jsweb-scraping
khurrum qureshi
khurrum qureshi

Reputation: 925

Scraping a website which requires authentication using node.js

I am trying to scrap this website https://www.erobertparker.com/entrance.aspx it requires authentication I am using request module to get authenticated like this,

 request({
        url:"https://www.erobertparker.com/login.aspx",
        method:"POST",
        form:{UNENTRY:"username",PWENTRY:"password"}
    },
    function(error,response,body){
 })

but i am unable to get authenticated what i am doing wrong can someone please guide me I am new to web scraping world :).

Upvotes: 3

Views: 11269

Answers (2)

Daniel César Indiano
Daniel César Indiano

Reputation: 9

Hi I solved this using a jar parameter in the request:

var j = request.jar();
    var request = request.defaults({ jar : j }) //it will make the session default for every request
    //...
    request({
        url:"https://www.erobertparker.com/login.aspx",
        method:"POST",
        form:{UNENTRY:"username",PWENTRY:"password"}
    },
    function(error,response,body){
        //Do your logic here or even another request like
        request({
            url:"<ANOTHER LINK>",
            method:"GET",
        }, function(error, response, body){
            //Some logic
        });
    });

You can also check the documentation of the request module: https://github.com/request/request#examples

Upvotes: 0

AndyD
AndyD

Reputation: 5385

It's using an asp.net session cookie. You possibly need to store all cookies in a jar and then send them back on the next request.

Upvotes: 1

Related Questions