Reputation: 4627
This is my first time on project that uses quite a bit of memory and I have observed a strange phenomenon.
I am reading in an excel file of about size 43MB
and saving it in an array. I think I would probably need to look for a way to stream to the client but I'm still curious.
I run my code by running node excelParserTool.js --max-old-space-size=10000
to allocate 10GB
of memory to my program.
The below is my code
let csvWriter = () => {
let workbook = new Excel.Workbook();
console.log(`before reading ${JSON.stringify(process.memoryUsage())}`);
workbook.xlsx.readFile('myfile').then((err, data) => {
console.log(`after reading ${JSON.stringify(process.memoryUsage())}`);
workbook.eachSheet((worksheet, sheetId) => {
let writeStream = fs.createWriteStream(`./sheet${sheetId}`);
worksheet.eachRow({includeEmpty: true}, (row, rowNumber) => {
let eachRow = '';
row.eachCell({includeEmpty: true}, (cell, colNumber) => {
if(cell.value !== null && typeof cell.value === 'object'){
eachRow += JSON.stringify(cell.value) + ', ';
}else if(cell.value === null){
eachRow += ', ';
}else {
eachRow += cell.value + ', ';
}
console.log(`through each cycle ${JSON.stringify(process.memoryUsage())}`)
});
eachRow += '\n';
writeStream.write(eachRow);
});
writeStream.end();
})
})
};
The point is that for each line that is read, I am printing process.memoryUsage()
so that I can see how much memory is being consumed. But when the program dies saying Javascript heap out of memory
the last memory usage says through each cycle {"rss":1549365248,"heapTotal":1509969920,"heapUsed":1479087128,"external":3724116}
Heaptotal and heap used is only about 1.5Gb so way below what I have allocated for the program. Also, when I see my computer stats, more than 7gbs are available still to be used. Why is this?
Upvotes: 0
Views: 882
Reputation: 8070
You set memory flag in wrong place
node [options] [ -e script | script.js ] [arguments]
so it should be node --max-old-space-size=10000 excelParserTool.js
in your case
Upvotes: 2