CODEWITHSUNDEEP
javascriptnode.jsstreamstreaming
Ohad Yeahhh
Ohad Yeahhh

Reputation: 321

Other purposes of data streaming besides videos/audio

I'm trying to understand the purposes of streaming (chiefly from a server to a visual client like a browser or an app). The basic concept of sending chunks of data instead of waiting for all the data to arrive is clear to me. I can easily think of use cases like loading audio or video, or maybe even photos. But are there any other use cases e.g. for sending textual data or jsons?

Upvotes: 1

Views: 31

Answers (1)

tam.teixeira
tam.teixeira

Reputation: 863

I've have used streaming to send all the records from a mysql table. So imagine there are millions of records and you want to extract all the records into an csv file. In this use case you cannot do a raw findAll without any filter/limit/offset, because it will try to take all the records into memory and when the table gets bigger it won't fit.

So instead of one findAll, i did several findAll's each returning about 50k records, for that i've used limit combined with offset. After each query i sent that chunk of 50k records into the stream and then query again for another 50k records. Did this in a recursive way until there are no more records (controlled this with the limit and offset). I had an rest endpoint, so when a user hit the endpoint it downloads that .csv file with all the table records.

I've found a article about it with postgres: https://medium.com/geoblinktech/how-to-export-a-postgresql-table-as-csv-with-node-js-streams-578d53434e80 (this uses the COPY operator)

Upvotes: 1

Related Questions