CODEWITHSUNDEEP
node.jsmemorymemory-management
forJ
forJ

Reputation: 4627

How to make NodeJS use all the memory allocated

This is my first time on project that uses quite a bit of memory and I have observed a strange phenomenon.

I am reading in an excel file of about size 43MB and saving it in an array. I think I would probably need to look for a way to stream to the client but I'm still curious.

I run my code by running node excelParserTool.js --max-old-space-size=10000 to allocate 10GB of memory to my program.

The below is my code

let csvWriter = () => {
    let workbook = new Excel.Workbook();

    console.log(`before reading ${JSON.stringify(process.memoryUsage())}`);
    workbook.xlsx.readFile('myfile').then((err, data) => {
        console.log(`after reading ${JSON.stringify(process.memoryUsage())}`);

        workbook.eachSheet((worksheet, sheetId) => {
            let writeStream = fs.createWriteStream(`./sheet${sheetId}`);

            worksheet.eachRow({includeEmpty: true}, (row, rowNumber) => {
                let eachRow = '';

                row.eachCell({includeEmpty: true}, (cell, colNumber) => {

                    if(cell.value !== null && typeof cell.value === 'object'){
                        eachRow += JSON.stringify(cell.value) + ', ';
                    }else if(cell.value === null){
                        eachRow += ', ';
                    }else {
                        eachRow += cell.value + ', ';
                    }
                    console.log(`through each cycle ${JSON.stringify(process.memoryUsage())}`)
                });

                eachRow += '\n';
                writeStream.write(eachRow);
            });
            writeStream.end();
        })
    })
};

The point is that for each line that is read, I am printing process.memoryUsage() so that I can see how much memory is being consumed. But when the program dies saying Javascript heap out of memory the last memory usage says through each cycle {"rss":1549365248,"heapTotal":1509969920,"heapUsed":1479087128,"external":3724116}

Heaptotal and heap used is only about 1.5Gb so way below what I have allocated for the program. Also, when I see my computer stats, more than 7gbs are available still to be used. Why is this?

Upvotes: 0

Views: 882

Answers (1)

ponury-kostek
ponury-kostek

Reputation: 8070

You set memory flag in wrong place

node [options] [ -e script | script.js ] [arguments]

so it should be node --max-old-space-size=10000 excelParserTool.js in your case

Upvotes: 2

Related Questions