CODEWITHSUNDEEP
javatomcattomcat7fiwarefiware-orion
Pedro
Pedro

Reputation: 449

Proton CEP: 100% CPU usage after a few hours

I have a Proton CEP instance deployed on my own server with 2 CPUs and 4GB RAM.

After leaving it working overnight, CPU usage increases heavily, up to 100% of each core. The command being executed is:

java -Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true -Xmx512m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Looking at the logs, I also see:

Feb 17, 2016 10:00:00 AM com.ibm.hrl.proton.server.executorServices.SimpleThreadFactory$ProtonExceptionHandler uncaughtException
SEVERE: Uncaught exception in thread: Thread[31516,5,main],exception: Java heap space
Feb 17, 2016 10:02:31 AM com.ibm.hrl.proton.server.executorServices.SimpleThreadFactory$ProtonExceptionHandler uncaughtException
SEVERE: Uncaught exception in thread: Thread[31643,5,main],exception: Java heap space

Althought from htop it seems that up to 2 GB of RAM are still free.

Is this normal?

The same server is also running Orion, but that one is not experiencing issues.

Upvotes: 2

Views: 162

Answers (1)

Pedro
Pedro

Reputation: 449

The problem was that due to a bug in our code, we were sending ever larger requests with data to Orion which forwarded them to Proton. Eventually the requests grew from 100B to over 50MB, causing Proton to stall as it was unable to process all the data in time.

Upvotes: 1

Related Questions