CODEWITHSUNDEEP
node.jsweb-scrapingrequest
Candleshine
Candleshine

Reputation: 1

Redirected request returns 400 Bad Request status

I am using Node's JS HTTP core request method with some mix of options from this module: https://github.com/request/request

I would like data from booking form on my website to be send to car park provider website and then I would like to scrap data about costs of car parks from a car parks booking site to display it on mine.

I am using following request:

   var form = {
        'TextBox_DateFrom': '16/10/15',
        'Arrive_Time': '32400',
        'TextBox_DateTo': '22/10/15',
        'Return_Time': '64800',
        'TextBox_PromoCode': '',
        'Button1': 'Get My Quote'
    };

    var formData = querystring.stringify(form);
    var contentLength = formData.length;


    var options = {
        followAllRedirects: true,
        maxRedirects: 20,
        hostname: 'directparking.co.uk',//'5.77.51.55:443',
        host: '5.77.51.55',
        path: '/index.php',
        port: '443',
        method: 'POST',
        headers: {
            'Content-Length': contentLength,
            'Content-Type':'application/x-www-form-urlencoded',
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36'
          }
    };

    var r = https.request(options, function(response){
        console.log('STATUS: ' + response.statusCode);
        console.log('HEADERS: ' + JSON.stringify(response.headers));
    });
    r.on('error', function(e) {
        console.log('error: ' + e);
        console.log('problem with request: ' + e.message);
    });
    r.write(formData);
    r.end();

That's what I get in return. I have no idea why there is 400 Bad request error within the Get method. You can have a look at the results from Node Inspector below:

400 Bad Request print screen

Upvotes: 0

Views: 753

Answers (1)

Aminah Nuraini
Aminah Nuraini

Reputation: 19156

I tried curling the page (directparking.co.uk) on my own. It needs you to curl it with cookie and user-agent at minimum. You don't use any cookie in your request or your request header.

Every web page has different minimum requirements to be scrapped. Some has a lot of requirements, some don't have any.

Upvotes: 1

Related Questions