CODEWITHSUNDEEP
javascriptnode.js
Mdlc
Mdlc

Reputation: 7288

Function to check if externally hosted textfile contains string acting slow- Node

I have the following code, which will retrieve a text file from an external server, and search the file for a specific string.

The function:

function checkStringExistsInFile(String, cb) {
var opt = {
    host: 'site.com',
    port: 80,
    path: '/directory/data.txt',
    method: 'GET',
};

var request = http.request(opt, function(response){
    response
    .on('data', function(data){
        var string = data+"";
        var result = ((string.indexOf(" "+ String +" ")!=-1)?true:false);
        cb(null, result);
    })
    .on('error', function(e){
        console.log("-ERR: Can't get file. "+JSON.stringify(e));
        if(cb) cb(e);
    })
    .on('end', function(){
        console.log("+INF: End of request");
    });
});
request.end();
}

And this is where I call the function, and do something with the results.

checkStringExistsInFile(String, function(err, result){
    if(!err) {
        if(result) {
            //there was a result
        } else {
            //string not present in file
        }
    } else {
        // error occured
    }
});

This worked great in the beginning (small text file), but my textfile is getting larger (4000 characters+) and this is not working anymore.

What can I do to solve this? Should I safe the temporary save the file on my server first, should I open the file as a stream?

It would be appreciated if you can support your answer with a relevant example. Thanks in advance!

Upvotes: 0

Views: 68

Answers (1)

jillro
jillro

Reputation: 4559

Documentation :

If you attach a data event listener, then it will switch the stream into flowing mode, and data will be passed to your handler as soon as it is available.

If you just want to get all the data out of the stream as fast as possible, this is the best way to do so.

http://nodejs.org/api/stream.html#stream_event_data

Data event is emitted as soon as there are data, even if the stream is not completely loaded. So your code just look for your string in the first lines of your stream, then callbacks.

What to do ?

Your function should only call callback on the end() event, or as soon as it finds something.

function checkStringExistsInFile(String, cb) {
  var opt = {
    host: 'site.com',
    port: 80,
    path: '/directory/data.txt',
    method: 'GET',
  };

  var request = http.request(opt, function(response){
    var result = false;
    response
    .on('data', function(data){
        if(!result) {
            var string = data+"";
            result = (string.indexOf(" "+ String +" ")!=-1);
            if(result) cb(null, result);
        }
    })
    .on('error', function(e){
        console.log("-ERR: Can't get file. "+JSON.stringify(e));
        if(cb) cb(e);
    })
    .on('end', function(){
        console.log("+INF: End of request");
        cb(null, result)
    });
  });
  request.end();
}

Upvotes: 1

Related Questions