CODEWITHSUNDEEP
javascriptnode.jssocketstcp
Jason Chen
Jason Chen

Reputation: 2577

NodeJS buffer incomplete TCP stream data

I am trying to identify an issue with my TCP JSON stream on my live server. What I have found is that if data streamed to me via TCP (in JSON format) is too large, then it does not consistently go through for parsing. I have to stream it a few times to be successful.

The code I am using goes as follows:

socket.once('data', function(data){
    let chunk = "";
    chunk += data.toString(); // Add string on the end of the variable 'chunk'
    let d_index = chunk.indexOf(';'); // Find the delimiter
    // While loop to keep going until no delimiter can be found
    while (d_index > -1) {         
        try {
            let string = chunk.substring(0,d_index); // Create string up until the delimiter
            let json = JSON.parse(string); // Parse the current string
            pages.addJSON(string);
            console.log(json.pagename); // Function that does something with the current chunk of valid json.    

        }
        catch(e){
            console.log(e);
        }
        chunk = chunk.substring(d_index+1); // Cuts off the processed chunk
        d_index = chunk.indexOf(';'); // Find the new delimiter
    }      
});

My data is streamed to me as a JSON file, with a ; to separate each stream. So for example, {"page": "something"};

I am using code from this question and one of the responses warns that we should buffer whatever code is not complete. I am wondering how to go about doing so, as I believe that my problem may stem from this issue.

Since my JSON stream is somewhat large, I think that a lot of the data does not pass through on connection, then gets cleared by my chunk variable.

The user stated that one possibility was to capture the data via byte size. Unfortunately, this option is not available to me because I do not know how large my streams are going to be.

I have used ; to capture the data's endpoint, and that's as far as I know what to do.

Upvotes: 1

Views: 1892

Answers (1)

jfriend00
jfriend00

Reputation: 707328

socket.once('data', ...) can give you problems because of the .once(). If you don't get ALL your data on the first data event, you will never see the rest of it. There are no guarantees at all about how much data comes on a given data event. It can vary depending upon a bunch of factors. On a slow link or a link with a proxy somewhere, you might get smaller pieces of data in multiple data events and if the data is large, you're probably almost guaranteed to get multiple data events as it will fill up a buffer somewhere.

You need to use socket.on('data', ...) so you can see all the data events in this transaction and then, if you want the event handler removed when you're done, you can then remove it when you've read everything and are done.

Also, note the code you borrowed from that other question uses socket.on(), not socket.once().

You also need to move the definition of the chunk variable outside the event handler so it survives from one event to the next.

let chunk = "";
socket.once('data', function(data){
    chunk += data.toString(); // Add string on the end of the variable 'chunk'
    let d_index = chunk.indexOf(';'); // Find the delimiter
    ....

Upvotes: 2

Related Questions