node js express: invalid request data?

Question

Could someone please explain to me the reason of the following behavior?

I have a nodejs/express webserver and let's say that one of the post routes containing the following code:

req.on('data', function() { console.log('data arrived'); })

This code works as expected and writes the 'data arrived' message to console when invoked. The following code however:

setTimeout(function() { req.on('data', function() { console.log('data arrived'); }) }, 1000);

doesn't print anything despite having the same circumstances. What exatly happens to the request object stream if we subscribe to their events after a short period of time?

The reason why I need this is that I'd like to upload large files to my server. When a post request arrives, I need to create a database entry and pipe my data into it. I cannot store data chunks in memory because of their potential sizes, but I cannot have the data of the request object from the database callback - or from any callbacks either.

Do you have an idea how to solve this issue? Thanks in advance!

UPDATE

The following code still doesn't work:

req.pause();
setTimeout(function() {
    req.on('data', function(data) { console.log('data arrived'); })
    req.resume();
}, 1000);

UPDATE2

The final goal would be to do sg like this:

req.pause();
doSometingAsync(function(result) {
    req.on('data', function(data) { doSomethingWithData(data, result) });
    req.resume();
});

, but this seems not to work...

UPDATE 3

This results in a single "elapsed" without anything else:

req.pause()
setTimeout () ->
    console.log 'elapsed'
    req.on 'error', (error) -> console.log error
    req.on 'data', (data) -> console.log 'in data'
    req.on 'end', () -> console.log 'end'
    req.resume()
, 2000

Answer 1

Answer:

you can fire req.pause(); when data arrives, handle the data like writing it into a file and when the file write callback fires you can do req.resume(); in order to receive the next package.

what you should not do is redefine the "data" event unless you used req.once instead of req.on

this would limit the bandwidth to your physical harddrive speed, would keep your app asyncronous and would lower ram usage.

UPDATE 1 Answer:

this here is some pseudo code:

req.on("data", function (data) {
    req.pause();
    /* do something with data like the following: */
    some_method(some_value, data, function (success) {
        if (success) {
            req.resume();
        }
    });
});

or something like this:

req.on("data", function (data) {
    req.pause();
    console.log("data arrived");
    setTimeout(function(){req.resume();}, 1000);
});

UPDATE 2 Answer: you should use a callback

in this case the third argument of doSomethingWithData(); to fire req.resume(); as soon as this function is done.

besides you should call the callback like this to stay async:

var doSomethingAsync = function (data, result, cb) {
    /* do something with data and result */
    process.nextTick(function () {
        cb();
    });
};

req.pause();
doSomethingAsync(function (result) {
    req.on('data', function (data) {
        req.pause();
        doSomethingWithData(data, result, function () {
            req.resume();
        });
    });
    req.resume();
});

or you do this:

var doSomethingAsync = function (data, result, req) {
    /* do something with data and result */
    process.nextTick(function () {
        req.resume();
    });
};

req.pause();
doSomethingAsync(function (result) {
    req.on('data', function (data) {
        req.pause();
        doSomethingWithData(data, result, req);
    });
    req.resume();
});