CODEWITHSUNDEEP
node.jsfirebasefirebase-realtime-database
Bassam
Bassam

Reputation: 97

Memory Leak in Firebase with Nodejs

I'm having a problem with Firebase when trying to load orders (20K nodes) by using the "child_added" event. Memory consumption increases and never goes down again. Once I stop the nodejs script, it goes back to normal.

My problem is that I'm running this script on a nodejs server (at startup), so, the memory on the server goes up and never goes down again.

tried to generate heap snapshot of memory after retrieving all children of the "orders" node. It showed that most memory allocation is happening inside objects:

which are basically members of the "sOrder" Firebase snapshot object. My original code was running without using "global.gc", but I added that to make sure Garbage Collector is running and in both cases, the same memory problem is happening.

I use the script like this:

node --expose-gc orders_load.js

I don't use debug to reproduce this case. My sample code:

console.log( "starting... " );
try
{
    let i = 1;
    firebase.database().ref("orders").on( "child_added", ( sOrder ) =>
    {
        console.log( "** " + i++ );
        if ( global.gc )
        {
            if( i % 1000 == 0 )
            {
                console.log( "running garbage collector" );
                global.gc();
            }
        }
        else
            console.log( 'Garbage collection unavailable.' );
    } );
}
catch( err )
{
    console.log( `EXCEPTION-%s`, err.message );
}

my "orders" node has 20K records and once I run the code above, the free memory goes down from 2181 Mb to 1426 Mb (755 Mb of memory). Any idea why the snapshot object of all my orders seem to be kept in memory even after callback function for the "child_added" event is finished?

firebase-admin: 6.4.0 nodejs: v11.6.0

=========================== After applying some hints from Frank below, I ended up with this code:

let i = 1;

function child_added( sOrder )
{
    console.log( "** " + i++ + " %s", sOrder.key );
}

console.log( "starting... " );
try
{
    firebase.database().ref("orders").once( "value", ( sOrders ) =>
    {
        sOrders.forEach( child_added );

        process.nextTick( () =>
        {
            firebase.database().ref("orders").limitToLast(1).on( "child_added", ( sOrder ) =>
            {
                child_added( sOrder );
            } );
        } );
    } );
}
catch( err )
{
    console.log( `EXCEPTION-%s`, err.message );
}

This works and memory consumption is gone. So, I'm receiving all children of "orders" node + receiving new orders coming + memory is low yay!!

Upvotes: 0

Views: 1928

Answers (1)

Frank van Puffelen
Frank van Puffelen

Reputation: 598728

When you attach an on listener, the listener stays active until you remove it with off. While the listener is active, Firebase will keep an in-memory copy of all data that the listener receives. The memory usage of this in-memory copy/cache can be quite significant, so you should remove your listeners when you don't need the data anymore.

To remove all listeners on a reference:

firebase.database().ref("orders").off();

Upvotes: 3

Related Questions