CODEWITHSUNDEEP
node.jshttpgetresponse
user3808357
user3808357

Reputation: 241

Nodejs Http request has no response

Currently using http GET to an external API. When called individually, the response is good. When put in a for loop, some requests don't seem to have a response.

This is the http GET function:

function httpGetChunk(url, callback) {
    http.get(url, function(resp) {
        var body='';
        resp.on('data', function(chunk) {
            body += chunk; //chunk too large from this response
        });
        resp.on('end', function() {
            var data = JSON.parse(body);
            callback(data);
        });
        resp.on("error", function(e) {
            console.log("Got error: " + e.message);
        });
    });
}

When I call the GET function in a for loop for 5 different urls, I only get responses for some of them. Ran it a couple of times and the response would be from a different combination of the called urls but never all of them.

Any insight?

Edit 1: To give more information, my for loop looks something like this.

for (var i=0;i<5; i++) {
    httpGetChunk(someUrl, function(data) {
        console.log(data);
    });
}

This would only print out some responses but not all.

Edit 2: I've taken into account all the advice on this thread. I'm now using the async module and have increased the number of concurrent connections to 20:

http.globalAgent.maxSockets = 20;

Following code is the one im currently testing:

getMatchStats() returns an game 'match' object with statistics (e.g kills, deaths in the match etc.)

matchIds is the array containing all the id keys of the matches

async.parallel([
    getMatchStats(matchIds[0], function (matchData) {
        console.log('0');
    }),
    getMatchStats(matchIds[1], function (matchData) {
        console.log('1');
    }),
    getMatchStats(matchIds[2], function (matchData) {
        console.log('2');
    }),
    getMatchStats(matchIds[3], function (matchData) {
        console.log('3');
    }),
    getMatchStats(matchIds[4], function (matchData) {
        console.log('4');
    }),
], function(err, result) {
    console.log('done');
    callback(result);
});

and getMatchStats

function getMatchStats(matchId, callback) {
    var url = getMatchStatsUrl(matchId); //gets url based on id
    httpGetChunk(url, function(data) {
        callback(data);
    });
}

again, the async.parallel never finishes since only some of the requests have responses. Every time i run it, the responses would be from a different combination of matches. Sometimes, it even completes all of the requests.

Maybe my OS has limitations on number of connections (im testing on localhost)?

Upvotes: 1

Views: 2984

Answers (1)

Rodrigo Medeiros
Rodrigo Medeiros

Reputation: 7862

Each request is asynchronous. So, if you use a regular for loop, each step is going to be executed synchronously and won't wait for callback to be called. What do you need is something like the each method from the async module, like:

async.each(yourArrayOfUrls, function (url, callback) {
  httpGetChunk(url, function(data) {
    console.log(data);
    callback();
  });
}, function (err) {
  // if some step produce an error, you can get it here...
});

Upvotes: 1

Related Questions