Java HashSet and HashMap JBoss-RMI Serialization exception with OpenJDK-1.8.0-312

Question

Since I installed openjdk-1.8.0-312 patch I got this error in all JBoss-RMI invocations that use HashMap and HashSet.

This is using openjdk-1.8.0 and running JBoss as 6.1.1:

2022-05-05 10:30:19,761 ERROR [STDERR]  ... 100 more
2022-05-05 10:30:19,761 ERROR [STDERR] Caused by: org.jboss.serial.exception.SerializationException: Excepted to be String
2022-05-05 10:30:19,761 ERROR [STDERR]  at org.jboss.serial.objectmetamodel.DataContainer$DataContainerInput.readUTF(DataContainer.java:1120)
2022-05-05 10:30:19,761 ERROR [STDERR]  at org.jboss.serial.persister.ObjectInputStreamProxy.readUTF(ObjectInputStreamProxy.java:196)
2022-05-05 10:30:19,761 ERROR [STDERR]  at org.jboss.serial.objectmetamodel.FieldsContainer.readField(FieldsContainer.java:147)
2022-05-05 10:30:19,761 ERROR [STDERR]  at org.jboss.serial.objectmetamodel.FieldsContainer.readMyself(FieldsContainer.java:218)
2022-05-05 10:30:19,761 ERROR [STDERR]  at org.jboss.serial.persister.ObjectInputStreamProxy.readFields(ObjectInputStreamProxy.java:224)
2022-05-05 10:30:19,761 ERROR [STDERR]  at java.util.HashSet.readObject(HashSet.java:298)
2022-05-05 10:30:19,763 ERROR [STDERR]  ... 104 more
2022-05-05 10:30:19,763 ERROR [STDERR] Caused by: java.lang.ClassCastException

Has anyone gotten this error? Any clue on how to solve this (without removing HashSet and HashMap from methods)?

Thanks

Answer 1

I know I'm two years late to this one, but I just spent days figuring this exact error out so here we go. I have an ancient JBoss EAP 5.x server I have to keep running long enough for the dev team to finish moving services out of it. (Why yes, I do work in enterprise, how could you tell?)

The root cause here is JBoss Serialization. Back in the Java 4-ish days JBoss created their own serialization implementation to be quicker than the one in the JVM. This worked fine until security fixes to deserialization appeared in later versions of Java 8, which broke the library. This is why even EAP 5.2.0 and 6.x break despite being originally built for Java 8, and why rolling back to earlier patch versions of the JVM works.

JBoss Serialization isn't really needed any more IMO, so the easy fix here is to disable it. In /deploy/ejb3-connectors-bean.xml, find the invokerLocator property of RemotingConnector and add the serializationType parameter to the invocation URL, like this:

  <bean name="org.jboss.ejb3.RemotingConnector"
    class="org.jboss.remoting.transport.Connector">

    <property name="invokerLocator">

      <value-factory bean="ServiceBindingManager"
        method="getStringBinding">
        <parameter>
          jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
        </parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>socket://${jboss.bind.address}:${port}?timeout=300000&amp;invokerDestructionDelay=5000&amp;serializationType=java</parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>3873</parameter>
      </value-factory>

    </property>
    <property name="serverConfiguration">
      <inject bean="ServerConfiguration" />
    </property>
  </bean>

There's a similar definition in /deploy/remoting-jboss-beans.xml. I'm pretty sure that one's not used any more, but no harm in adding the parameter there too:

<bean name="UnifiedInvokerConfiguration" class="org.jboss.remoting.ServerConfiguration">
   <constructor>
      <!-- transport: Others include sslsocket, bisocket, sslbisocket, http, https, rmi, sslrmi, servlet, sslservlet. -->
      <parameter>socket</parameter>
   </constructor>
 
 
   <!-- Parameters visible to both client and server -->
   <property name="invokerLocatorParameters">
      <map keyClass="java.lang.String" valueClass="java.lang.String">
 
         <!-- Other map entries ... -->
          
         <entry><key>serializationType</key><value>java</value></entry>
 
      </map>
   </property>
    
</bean>

That should disable JBoss Serialization for all remote EJB invocations. However, I did find one more issue with Interceptors. The jboss-ejb3-core library contains two AOP interceptors that are designed to check if a service is accidentally calling a remote binding that is provided locally, and turn that call into a local invocation instead. Good idea on the surface, but these interceptors do not have an equivalent of the serializationType parameter. They are hard-coded and will always use JBoss Serialization. Personally I consider this a bug, but JBoss 5.x ain't getting patches any time soon!

There are two pretty simple options to work around this. You can of course update your services to use local bindings. That would be the preferred option, but obviously as a server admin it's hard to anticipate where these issues are present and we don't want to go breaking things that currently work.

The second option is that you simply disable the interceptors in /deploy/ejb3-interceptors-aop.xml. I'm not sure I can recommend doing this, as I've had issues with it. It will have a negative performance impact on any service that incorrectly uses its own remote bindings, as those calls will actually get dispatched to the remoting server. It can also mess with transactions across services. My initial tests with disabling the interceptor resolved the initially observed bug in one service, but immediately caused transactional errors in another. I have a sinking feeling I'm going to end up building my own version of jboss-remoting-aspects or something further down the execution stack and force usage of JVM serialization.

Update 2025-01-13: I did actually patch jboss-ejb3-core, editing the IsLocalInterceptor to use the SerializationManagerFactory to fetch an appropriate serializer based on server settings, as works elsewhere. However, this led to one final complication: JBoss Serialization doesn't have the same strict requirements as serialization in the JDK. For one thing, it doesn't require you to implement Serializable. After over a decade, I can't imagine the amount of detection and testing that would need to be done to correct all of these. Long story short, we're biting the bullet and going through to replace any unnecessary remote calls with local ones, so that the interceptor never fires in the first place.

Obviously this post references JBoss 5.x but hopefully it gives anyone reading this a starting point for where to look on later versions. I found a number of instances of people running into this issue around the net, but no concrete fixes, so as late as the reply is I hope it's still of use to someone.

Answer 2

Try switching back to previous versions of openjdk.