Coldfusion server error - Corrupt form data: premature ending

Question

I've got a problem with our live server. Occasionally a page will not load, and the following error is displayed in plain black text on a white background:

 Server Error
 The server encountered an internal error and was unable to complete your request
 Application server is busy. Either there are too many concurrent requests

If I look in the error log for the website, at the time of the error using the following command:

 grep "12:17:33" error_log

lines such as this are returned:

 [Tue May 03 12:17:33 2011] [notice] jrApache[14352: 21512]  returning error page for JRun too busy or out of memory

Then if I search in the Coldfusion cfserver.log I find these errors, a few seconds before hand:

05/03 12:17:29 error Corrupt form data: premature ending java.io.IOException: Corrupt form data: premature ending at com.oreilly.servlet.multipart.MultipartParser.(MultipartParser.java:177) at com.oreilly.servlet.multipart.MultipartParser.(MultipartParser.java:99) at coldfusion.filter.FormScope.fillMultipart(FormScope.java:190) at coldfusion.filter.FusionContext.SymTab_initForRequest(FusionContext.java:369) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:33) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:53) at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

05/03 12:17:29 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@500b675]) JRunPRoxyServer.invokeRunnable: java.lang.IllegalStateException at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205) at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

java.lang.IllegalStateException at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205) at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

I don't really know what to do from here.

If the page is refreshed then it loads without error. I don't know how to replicate the error, it doesn't seem to happen on any particular page or at any articular time.

I've tried setting:

 Maximum number of simultaneous Template requests: 100

in the Coldfusion administrator, I've done that based on Steven Erat's post: http://www.talkingtree.com/blog/index.cfm/2005/3/11/ContextSwitchingBad

But I'm still getting errors. Would increasing this to a higher value help?

And then there's the "premature form ending" part of the longer Java error, I'm not sure if that's related and don't understand what it means anyway.

I've also looked through my access logs, and ruling out any images or js files that get called at the time of any errors, I've got a variety of CFM pages that return a 503 http status, and some of them are CFM pages that get returned with a content type of Javascript, and are never requested after a form post.

So in summary, I'm getting some pretty blunt error messages, and I can reproduce them at present, and the page loads fine if I just hit F5, and I'd like to know if anyone can help me to understand the source of the problem.

Oh and one last thing, we're running a RHEL 5 server with CF9, and we've upgraded recently from CF7. Before the upgrade this error was not happening.

Answer 1

I'm really sorry to answer this myself, and I'm not entirely sure what fixed it.

But I installed FusionReactor, which required me updating my JVM, and then rebooting coldfusion. Since then, the problem has not occurred.

I was getting it daily, at least hourly but at peak times a couple of times a minute. Now it hasn't happened in 5 days.

I'd love to hear any thoughts on why any of these might have fixed it?

Answer 2

This is just a guess from my testing of CF 9 to solve a problem we are having, but it may just be the amount of RAM you have (and how much it is being used up).

I found that in CF 9, if you hit your memory limit, it will kill your JVM process automagically and restart it so all your memory is freed up. (we have a problem with our older version of CF that just brings the server to its knees if the ram is maxed out). Do you have a test environment where you can put a load on the server and see what happens when you max your RAM?