ColdFusion - missing class file = JRun error

Question

My team runs ColdFusion (v9) on five separate servers. Three of them are on a load balanced router, and the other two are development and test machines.

Several months ago, we began to have issues where certain page/template requests would fail. When I say fail, I mean that the server would return a 500 error. The error, however, did not trigger our ColdFusion exception handling via cftry/cfcatch and cferror.

After examining the HTTP headers, it looked like some sort of jrun error, so I went into the exception log. Here is an example of an error (this one happened this morning):

"Error","jrpp-1","07/30/12","06:30:02",,"(class: cfezReporting2ecfc400556386, method: runPage signature: ()Ljava/lang/Object;) Incompatible object argument for function call The specific sequence of files included or processed is: X:\docs\ezBuilder\index.cfm'' "
java.lang.VerifyError: (class: cfezReporting2ecfc400556386, method: runPage signature: ()Ljava/lang/Object;) Incompatible object argument for function call
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
    at java.lang.Class.getConstructor0(Class.java:2699)
    at java.lang.Class.newInstance0(Class.java:326)
    at java.lang.Class.newInstance(Class.java:308)
    at coldfusion.runtime.TemplateClassLoader.newInstance(TemplateClassLoader.java:552)
    at coldfusion.runtime.TemplateClassLoader.newInstance(TemplateClassLoader.java:523)
    at coldfusion.runtime.TemplateProxyFactory.getCFCInstance(TemplateProxyFactory.java:270)
    at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:173)
    at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:158)
    at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:148)
    at coldfusion.cfc.ComponentProxyFactory.getProxy(ComponentProxyFactory.java:62)
    at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:373)
    at cfezBuilder2ecfc293785079$funcCREATE_UVN._factor7(X:\docs\ezBuilder\components\ezBuilder.cfc:376)
    at cfezBuilder2ecfc293785079$funcCREATE_UVN.runFunction(X:\docs\ezBuilder\components\ezBuilder.cfc:327)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:472)
    at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:405)
    at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:368)
    at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:55)
    at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:321)
    at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:517)
    at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2547)
    at coldfusion.tagext.lang.InvokeTag.doEndTag(InvokeTag.java:460)

Once this error occurred on a template, it would continue to occur, i.e. it was not sporadic and didn't 'fix itself'.

Another interesting tidbit is that this error seemed to occur on all the servers simultaneously on the same file. By simultaneously, I mean by the time we received notice that one is erroring, we test the other servers and find they are also erroring.

Not being able to find help on this particular error detail anywhere, I made a guess that somehow ColdFusion was unable to find a template or class file. I checked each server and the .cfc file was there (plus, the 'template not found' exception has always been different).

So instead, I made a change to the file (ezReporting.cfc in the above example) and copied it to all the servers, and lo and behold, it worked. The change was simply adding and then removing a space - basically forcing the template to recompile, I guess. I did this each time over the next few weeks each time the error occurred. It was always on that .cfc file.

The next time the error happened, about a week later, it was on the exact same file. This time, I cleared the template cache on the server rather than physically change the template. It worked again.

Since then, this same error occurred on several different files - both CFCs and CFM files. I did not notice a pattern in (a) the "age"/last updated date of the files - everything from a day old to over a year, (b) the content of the files - everything from simple blocks to SQL queries to some basic set/looping, or (c) the names of the files.

This morning, it occurred on yet another file, one that I knew worked two days prior on all servers. The CFC file contents had not been touched on any servers since last year. And, when I went to all five servers, the exact same file failed on all of them. When I cleared the template cache, every server started working again.

I'm giving all these boring details because I'm reaching for anything that can help. Perhaps there is some sort of expiration occurring on the file itself, and that would explain why the file expired on all servers right around the same time -- we change it on the development server and then move it out to the test and production servers with a simple copy/paste. But, there doesn't seem to be any rhyme or reason for expiration, because the file in question is of varying ages as noted above.

I tried the Adobe forums to get help on this particular exception/dump, but have not found anyone who has run into the same thing.

Does anyone else have any ideas? This error troubles me because it doesn't trigger our normal exception handling, so we can't do much about it without human intervention. Thanks for any specific guidance.

Answer 1

For anyone who has this same issue, I wanted to add a pseudo "solution" that I worked on recently. I say pseudo because it doesn't address the root problem which is still unknown to me. But it does allow the application to recover from the error.

In short, ColdFusion <cfcatch>, even with type="any", does not catch errors thrown from the JRE. However, this particular error can be caught with a <cfcatch type="java.lang.VerifyError">.

In that cfcatch, I use my administrator functions to clear the template cache with the clearComponentCache() and clearTrustedCache() methods. Then, rather than copy the code that just failed, I do a <cflocation> for the user to return the same URL to try again.

While forwarding the user works in this particular application, it could be problematic for situations such as ajax calls; however, in that case I think it would be possible to simply re-include the cfms or re-invoke the CFCs that errored the first time.

Let me know if this works for you or if anyone else has found a better solution.

Answer 2

I have had the same issue with and unfortunately was never able to fix it. The one space trick was exactly the same a tiny modification to for re-compilation fixed it every time.

Answer 3

As Barney says you a likley experiencing a corrupt template cache. Clearing out the files int the /cfclasses folder might be a good starting point - then check the health of your file system (defragging and all that). You may have file I/o problems loading the classes. It is these classes and not the .cfm file that are really being "run" by your web server.

The only thing that flumoxes me is that the error "occurs simultaneously". Do you have some sort of precompile process going on? Do all of the sites share the same physical codebase on an NAS or something? That's the part that seems odd to me.