CODEWITHSUNDEEP
javaramknime
Steve
Steve

Reputation: 11

Memory footprint of KNIME is growing until the machine freezes

Love KNIME and what it can do to standardise/simplify ML, but I’ve a new set of problems that are driving me crazy. I’m running ML analysis of tables of data with ~400 columns/400 rows. Non-trivial certainly, but not huge. I’ve a simple workflow that currently just evaluates each column, one at a time, as a feature, so no combinatorics.

I’m using a respectable linux machine with 16GB RAM. KNIME starts the analysis taking 10% of the RAM (according to Htop) and just grows and grows 0.1% at a time until eventually KNIME has consumed 78% of RAM and the whole of the RAM is full between KNIME and OS. Then the swap memory starts filling up. Once that reaches 100%,the machine then freezes.

I’ve tried adjusting Xmx in knime.ini and it has no effect whatsoever and I’ve added the heavy garbage collector node to my loop also to no effect.

Can anyone offer any help? Getting a bit desperate here! I’m running knime v4.6.

Thanks Steve.

Upvotes: 1

Views: 195

Answers (1)

Bryan
Bryan

Reputation: 21

It sounds like you're letting KNIME have more RAM than your machine can offer, which may be what's causing your computer to freeze. You mentioned adjusting Xmx setting, have you tried dialing it down, not just up?

Have you been able to run your workflow on a different machine, or has someone else been able to run the workflow? It's possible there is a something in the workflow that is causing you to eat up too much memory. If you share your workflow, that might help determine if there is a problem with it.

I also came across this interesting post in the KNIME forum, I wonder if it might help? https://forum.knime.com/t/best-memory-optimization-practices/32775

Upvotes: 0

Related Questions