CODEWITHSUNDEEP
javawso2
Lucas Rhode
Lucas Rhode

Reputation: 1

WSO2 IdendityServer throwing ThreadDeath Exception

I start the server using the ./wso2server.sh start command but after a few hours the server stops responding and throws a ThreadDeath exception.

I tried modifying the JVM_MEMOPTS environment variable to increase heap size JVM_MEM_OPTS="-Xms1024m -Xmx2048m" and that seemed to allow the server to run longer before the exception happens, but that may be just a coincidence honestly.

Do you have any ideas on how to fix this?

Stacktrace:

TID: [-1234] [authenticationendpoint] [2022-04-27 16:37:30,910] [86418802-68be-45af-af12-fd4bbfd54bf4] ERROR {org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/authenticationendpoint].[jsp]} - Servlet.service() for servlet [jsp] in context with path [/authenticationendpoint] threw exception [javax.servlet.ServletException: java.lang.ThreadDeath] with root cause java.lang.ThreadDeath
        at java.base/java.lang.Thread.stop(Thread.java:937)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.handleStuckThread(CarbonStuckThreadDetectionValve.java:111)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.backgroundProcess(CarbonStuckThreadDetectionValve.java:164)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1145)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)

INFO:

openjdk 11.0.15 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)

Upvotes: 0

Views: 308

Answers (1)

Sajith
Sajith

Reputation: 1269

According to the log, there are JSP files with random names.

  • /authenticationendpoint/28oc6hgokjawkyo9zhidiqzmie3.jsp

Therefore, your deployment seems to be compromised with the vulnerability [1]. These JSP files could be executing commands which are running for a longer time and causing the thread deaths.

You can follow the steps provided in the same document to mitigate the vulnerability.

But when a remote code execution vulnerability gets exploited, there are numerous ways a malicious actor could create backdoors within the system and establish persistence. Due to different attack vectors, and techniques used in the exploitation, it is not possible to provide steps that are usable to cleanup an affected deployment with an assurance that the malicious actor would not be able to regain access.

Therefore, the security recommendation is to recreate the instance (without reusing the VM/server instance). However, you may use the existing configuration files from the previous deployment, after reviewing that there are no unintended modifications made to the configuration files.

[1] https://docs.wso2.com/display/Security/Security+Advisory+WSO2-2021-1738

Upvotes: 1

Related Questions