Error in JBOSS deployment javax.management.InstanceNotFoundException

Question

I'm having an issue very strange in my enviroment wich is jboss-5.1.0.GA java version "1.6.0_33" Java(TM) SE Runtime Environment (build 1.6.0_33-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)

It was completely functional until I update a version of some jar (wich was tested in 6 enviroments before this one without the error) in the context. before the error i had this files:

server/default/deploy/myfolder

-rw-r--r--  1 jboss users  808430 May 16 17:29 s-c-core-2.38.6.0.jar
-rw-r--r--  1 jboss users  555328 May 16 17:29 s-c-dao-2.38.6.0.jar
drwxr-xr-x  9 jboss users    4096 Jun  3 20:11 s-c-web.war
-rw-r--r--  1 jboss users  199259 May  2 16:58 s-com-core-2.38.5.0.jar
-rw-r--r--  1 jboss users  202086 May  2 16:58 s-com-dao-2.38.5.0.jar
-rw-r--r--  1 jboss users 1333612 May  2 16:58 s-com-domain-2.38.5.0.jar
-rw-r--r--  1 jboss users   86526 May  2 16:58 s-com-util-2.38.5.0.jar
-rw-r--r--  1 jboss users  660621 May 11 13:01 s-fin-core-2.38.6.0.jar
-rw-r--r--  1 jboss users  559292 May 11 13:01 s-fin-dao-2.38.6.0.jar
drwxr-xr-x  8 jboss users    4096 May 15 21:41 s-fin-web.war
-rw-r--r--  1 jboss users  108444 May  7 07:29 s-geio-engine-2.38.5.0.jar
drwxr-xr-x  8 jboss users    4096 May 15 21:24 s-geio-web.war
-rw-r--r--  1 jboss users   71975 Jun 14  2012 s-o-core-1.1.4.0.jar
-rw-r--r--  1 jboss users   66516 Jun 14  2012 s-o-dao-1.1.4.0.jar
-rw-r--r--  1 jboss users   53532 Jun 14  2012 s-o-domain-1.1.4.0.jar
-rw-r--r--  1 jboss users   16347 Jun 14  2012 s-o-util-1.1.4.0.jar
drwxr-xr-x  5 jboss users    4096 Jun 14  2012 s-o-web.war
-rw-r--r--  1 jboss users   54588 Apr 10 11:29 s-p-core-2.38.2.0.jar
-rw-r--r--  1 jboss users   35784 Apr 10 11:29 s-p-dao-2.38.2.0.jar
drwxr-xr-x  8 jboss users    4096 Apr 10 19:20 s-p-web.war
-rw-r--r--  1 jboss users   69140 Jun 14  2012 s-s-se-1.1.0.jar
-rw-r--r--  1 jboss users  594214 May 13 13:36 s-ser-core-2.38.6.1.jar
-rw-r--r--  1 jboss users  530903 May 13 13:36 s-ser-dao-2.38.6.1.jar
drwxr-xr-x  9 jboss users    4096 May 15 21:58 s-ser-web.war
drwxr-xr-x  6 jboss users    4096 May 28 08:27 s-te-me.war
-rw-r--r--  1 jboss users  155639 May 15 22:06 s-uni-core-2.38.5.0.jar
-rw-r--r--  1 jboss users  145944 May 15 22:05 s-uni-dao-2.38.5.0.jar
drwxr-xr-x  7 jboss users    4096 May 15 21:20 s-uni-web.war

then I stop my jboss (/etc/init.d/jbossd stop) and update just theese three files/folder (deleting the old ones and putting the new ones)

-rw-r--r--  1 jboss users  808430 May 16 17:29 s-c-core-2.38.7.0.jar
-rw-r--r--  1 jboss users  555328 May 16 17:29 s-c-dao-2.38.7.0.jar
drwxr-xr-x  9 jboss users    4096 Jun  3 20:11 s-c-web.war

after server start with the user root and with the command "/etc/init.d/jbossd start" at some point the users start to use applications and on the log file server/default/log/server.log start to appear some strange errors like this:

2013-06-04 00:01:30,615 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jmx-console].[HtmlAdaptor]] (http-0.0.0.0-80-3) Servlet.service() for servlet HtmlAdaptor threw exception
javax.management.InstanceNotFoundException: jboss.j2ee:jar=s-c-core-2.38.6.0.jar,name=ApCarBusinessImpl,service=EJB3 is not registered.
      at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:526)
      at org.jboss.mx.server.MBeanServerImpl.getMBeanInfo(MBeanServerImpl.java:675)
      at org.jboss.jmx.adaptor.control.Server.getMBeanData(Server.java:98)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$1.run(HtmlAdaptorServlet.java:357)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$1.run(HtmlAdaptorServlet.java:354)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.getMBeanData(HtmlAdaptorServlet.java:353)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.inspectMBean(HtmlAdaptorServlet.java:224)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
      at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doGet(HtmlAdaptorServlet.java:81)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
      at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      at java.lang.Thread.run(Thread.java:619)

Saying that the EJBs are not instantiated (lots of then) but nothing happens to the user, the application keep working normally

So I start to verify if the file s-c-core-2.38.6.0.jar wasn't left behind in some folder inside my jboss and NO it didn't. So I start to think that is some kind of cache that the jboss was reading so I stop the service and delete the folders server/default/tmp and server/default/work and restart the server but the error keep going on the log.

My run.conf is with this configuration:

JAVA_OPTS="-DSB_BASE=$SB_BASE -Dmodulo.context.file=s-o-core-context.xml;s-com-core-context.xml;s-ser-core-context.xml;s-geio-core-context.xml;s-fin-core-context.xml;s-c-core-context.xml;s-uni-core-context.xml;s-p-core-context.xml -Xmx3072m -Xms768m -XX:MaxPermSize=256m -Xss1024k -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -Djava.awt.headless=true"

I tried a lot of things like user permissions, folder permissions, memory configurations (xmx xms) and the error keep appearing in the log file.

When I came back to the files that I changed ( s-c--2.38.6.0 ) the application stop to show this error, so I obviously though that the erro is with that package then I applyed it on a mirror server and the error does'nt show up.

So I am without options right now. The application is working fine for the user but the error keep appearing in the log and this is very annoying.

Has someone here experienced something like this?

I've posted this question on community.jboss.org 2 days ago with no help:

Thanks in advance, sorry about my english.

Answer 1

The error basically implies that JBoss was unable to find the service MBean in its tree. Do you see that service deployed in the jmx-console?

You are seeing the InstanceNotFoundException: jboss.j2ee:jar=s-c-core-2.38.6.0.jar,name=ApCarBusinessImpl,service=EJB3 is not registered because:

Some external service/tool/app or your own application is trying to invoke that particular service (name=ApCarBusinessImpl,service=EJB3). Since JBoss cant find it deployed anywhere (as a result of you updating the deployed JAR), it simply logs the message that it couldn't find the MBean that it was asked to invoke.

To resolve the issue you will need to find out who is making the request on the old service name (jboss.j2ee:jar=s-c-core-2.38.6.0.jar) and have it updated to point to the new service name. This should resolve seeing those exception log messages.

Edit by Jorge Campos:

As @Coolbeans said it was indeed a external service that was calling my beans started on JBoss. I found out that in the installed jboss was a security issue that was not properly configured (jmx-console). So, there was a worm that was making requests to a previosly installed beans in my application server (some kind of cache). After some digging inside the jboss installation I find this war iesvc.war which is a worm as stated in this thread https://community.jboss.org/blogs/mjc/2011/10/20/statement-regarding-security-threat-to-jboss-application-server

The worm is JAVA_JBOSSPY.A identified by a trend micro office scan.

So thank you for the answer @Coolbeans you were right it was a "Some external service/tool/app" as you said.

Answer 2

This is sometimes caused by the PU name. Try to change the PU name to another name, clean and build and deploy.

Regards