CODEWITHSUNDEEP
javamysqlamazon-web-servicescoldfusion-10ubuntu-13.04
PeterKA
PeterKA

Reputation: 24638

Java Running at close to 100% CPU

Too much logging activity over the weekend led to the following error being thrown by ColdFusion:

Message:    No space left on device
StackTrace: java.io.IOException: No space left on device at
            java.io.FileOutputStream.writeBytes(Native Method) at 
            java.io.FileOutputStream.write(FileOutputStream.java:269) at 
            coldfusion.compiler.NeoTranslator. ......

By this morning pages on the ColdFusion web site were not loading at all. The disc (12Gig) was over 99% used up. We moved several files to the second disc and now it's at about 80%, much lower than where it's always been. We're going to direct logging activity to the second disc (100Gig) to prevent a repeat. Having created space on the disc we restarted apache and coldfusion but still pages are not loading.

When we run top -H it appears that java is running at close to 100% CPU. Does anyone have a clue what's going on or what info I need to provide so someone can figure it out?

The set up is AWS, ubuntu 13.04, coldfusion 10, mysql (rds).

UPDATE

I have made some really strange but hopefully helpful observations. I am still trying to locate a tool to help with getting thread dumps. Whenever I restart ColdFusion, most pages load fine and CPU usage seems normal, mostly 0.7 - 1.5% but once in a while spikes of up to 10% can be seen. But there's one particular page, which when I try to load, causes CPU uses to rise to 97% always. The pages that load fine have a simple query reading data from one table. This problematic page has an inner join, and reads data from two tables. I don't know how helpful this is but I think it is too consistent to be insignificant.

UPDATE 2

And since this issue started the following errors appeared for the first time and there're hundreds of lines of it:

[2243:140630871263104] [error] ajp_send_request::jk_ajp_common.c (1649):
(cfusion) connecting to backend failed. Tomcat is probably not started 
or is listening on the wrong port (errno=111)

// and

[2234:140630871263104] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): 
Failed opening socket to (127.0.0.1:8012) (errno=111)

// and 

[2756:139822377088896] [info] jk_handler::mod_jk.c (2702): No body with status=500 
for worker=cfusion

UPDATE 3 - RESOLVED

After stripping the "offending" page of all the code in it and simply replacing it with some plain text and tried to load the page several times, we realized that ColdFusion was not loading the live pages. It was loading cached compiled versions of the pages, normally found in a subfolder named <cf-root>/cfusion/wwwroot/WEB-INF/cfclasses. Removing (or renaming) the subfolder resolved the issue.

Upvotes: 0

Views: 285

Answers (1)

Aaron
Aaron

Reputation: 24802

You would probably want to provide thread dumps, a few of them taken at a few seconds interval, and gc logs.

Thread dumps can be produced with jstack (a tool provided in your jdk's bin directory), and garbage collector logs must be activated beforehand.

Upvotes: 1

Related Questions