CODEWITHSUNDEEP
javascriptnode.jsexpressstreamsails.js
kk415kk
kk415kk

Reputation: 1247

Piping data from a file to a rendered page in Sails.js

My application needs to read in a large dataset and pipe it to the client to manipulate with D3.js. The problem is, on large datasets, the reading/loading of the file contents could take a while. I want to solve this using streams. However, I'm unsure of how to do so in the context of the Sails framework.

What I want to do is read the contents of the file and pipe it to a rendered page. However, I can't figure out how to pipe it through if I use something like res.view('somePage', { data: thePipedData });.

I currently have something like this:

var datastream = fs.createReadStream(path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv'));
datastream.pipe(res); 
...
return res.view('analytics', { title: 'Analytics', data: ??? });

What's the best way to approach this?

Upvotes: 3

Views: 1792

Answers (1)

sgress454
sgress454

Reputation: 24948

Based on your example it seems like the best course of action would be to set up a separate endpoint to serve just the data, and include it on the client via a regular <script> tag.

MyDataController.js

getData: function(req, res) {

    /* Some code here to determine datatype and dataset based on params */

    // Wrap the data in a Javascript string
    res.write("var theData = '");
    // Open a read stream for the file
    var datastream = fs.createReadStream(
        path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv')
    );
    // Pipe the file to the response.  Set {end: false} so that res isn't closed
    // when the file stream ends, allowing us to continue writing to it.
    datastream.pipe(res, {end: false});
    // When the file is done streaming, finish the Javascript string
    datastream.on('end', function() {
        res.end("';");
    });

}

MyView.ejs

<script language="javascript" src="/mydata/getdata?datatype=<%=datatype%>&etc.."></script>

MyViewController.js

res.view('analytics', {datatype: 'someDataType', etc...});

A slight variation on this strategy would be to use a JSONP-style approach; rather than wrapping the data in a variable in the data controller action, you would wrap it in a function. You could then call the endpoint via AJAX to get the data. Either way you'd have the benefit of a quick page load since the large data set is loaded separately, but with the JSONP variation you'd also be able to easily show a loading indicator while waiting for the data.

Upvotes: 5

Related Questions