CODEWITHSUNDEEP
javascriptnode.jsfirebasefirebase-realtime-databasestream
Alex Ironside
Alex Ironside

Reputation: 5039

Streaming data from Firebase Realtime Database

I have over 20k documents in my Realtime Database. I need to stream them, but I'm not even sure how to get started with it. This is kind of what I was trying to go for

sendEmail.get('/:types/:message', cors(), async (req, res, next) => {
    console.log(5);
    const types = JSON.parse(req.params.types);
    console.log('types', types);
    let recipients = [];
    let mails = [];
    if (types.includes('students')) {
        console.log(1);
        const tmpUsers = await admin.database().ref('Users').orderByChild('student').equalTo(true).once('value').then(r => r.val()).catch(e => console.log(e));
        recipients = recipients.concat(tmpUsers);
    }
    if (types.includes('solvers')) {
        console.log(2);
        let tmpUsers = await admin.database().ref('Users').orderByChild('userType').equalTo('person').once('value').then(r => r.val()).catch(e => console.log(e));
        tmpUsers = tmpUsers.concat(arrayFromObject(await admin.database().ref('Users').orderByChild('userType').equalTo('company').once('value').then(r => r.val()).catch(e => console.log(e))));
        recipients = recipients.concat(tmpUsers);
    }
});

But this code causes my server to run out of memory. Someone suggested streams in my previous question, but as much as I like the idea, I have no idea how to actually do the streaming. I know it should be something like:

const fs = require('fs');
const readStream = fs.createReadStream('path goes here');
const data = [];
readStream.on('data', chunk => {
    data.push(chunk);
})
readStream.on('end', () => {
    console.log(data);
    res.end(data);
});

But how on earth do I pass a firebase query into the path?

I tried this, but it said Argument type Promise is not assignable to parameter type PathLike, which makes sense, but how do I get around it?

const users = fs.createReadStream(admin.database().ref('News').once('value').then(r => r.val()))

To summarize: How do I stream data from a Firebase realtime database?

Edit: How is that a duplicate? These are 2 completely different questions. The starting code is the same but the ways of solving I'm asking about are completely different

Upvotes: 0

Views: 2270

Answers (1)

Doug Stevenson
Doug Stevenson

Reputation: 317372

Realtime Database doesn't have any streaming APIs. When you perform a query, the entire result set is always loaded into memory in one shot.

A possible alternative is to page through the results using limitToFirst() along with some query ordering that allows you to perform a subsequent query after the first batch of items is processed, picking up where the last query left off.

Upvotes: 4

Related Questions