Spring app consumes all CPU and then crashes

Question

I have a Spring application that has been recently moved from test to production environment. CPU usage is %2-3 most of the time. But sometimes (1 or 2 times per day) it suddenly peaks up to %400 (there are 4 cpus) and then my app crashes. I could not observe any kind of anomalies (e.g. extra traffic) when the problem occurred.

Some information about my app is,

• It has some REST endpoints that are called by android clients.
• It has 3 ScheduledTasks in one of the Services. They make some HttpUrlConnections.
• My traffic is not very huge, e.g. 300-400 requests per minute and it is mostly stable, i.e. no peaks are observed in network calls.
• CPU usage reaches a maximum value of %3 with this traffic, except the times that the problem occurs. Here is a visualization of the CPU utilization for the last 24 hours:
• My server has 4 core cpu, 12 GB RAM and 300GB Full SSD disk. Linux version 4.4.0-93-generic (buildd@lgw01-03) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) )
• Memory settings: -d64 -Xms6g -Xmx10g -XX:MaxPermSize=512m
• I used sysstat for logging. I am watching the cpu and ram usage not only for the whole system but also for my app only. I would like to note also that ram usage peaks up to max %60.
  An example from CPU logs GC logs and Thread Dumps: I logged GC, and also Thread dumps (jstack) with 2 minutes interval. No difference was displayed normal time, before crash and at crash time.
• Logs between : 20:30 - 06:26
• Server crash down time: 06:26
• GC Analysis
• Thread dumps ( Thread dump 1 (05:40) .. Thread dump 14 (06:24) ) ( Other Thread Dumps are before 05:40)
• GC analysis comment: "97.14% of the time it was due to Allocation Failure. But its time that 21:12:22 when 9 hours before crash."
• Last errors in log file
  • 2017-10-09 06:26:19.393 ERROR 5986 --- [ XNIO-2 task-1] t.o.m.w.rest.errors.ExceptionTranslator : An unexpected error occurred: I/O error while reading input message; nested exception is java.io.InterruptedIOException: XNIO000808: I/O operation was interrupted
  • 2017-10-09 06:26:19.396 ERROR 5986 --- [ XNIO-2 task-25] io.undertow.request : UT005022: Exception generating error page /error

I could not find the reason of the problem. Are there any suggestions? Any kind of help is appreciated.

Answer 1

As the invaluable comments indicate, this problem does not originate from garbage collection. The reason is something totally different. It took about a week to figure it out, so I am writing it to here in case some other people face with the same problem.

First of all, I did not include in the question (since I did not consider it as important) that I used JHipster, which recently switched to Undertow as the application server.

After digging into the details, I observed that the threads that consume the CPU are XNIO threads. I have seen on a few sites that some people using application servers powered by XNIO also suffer from the same problem. Since Undertow uses XNIO, I changed from Undertow to Tomcat, and the problem is resolved.