WebSphere JAX-WS Response Handler fails

Question

Scenario: We hava a 3rd party CRM application which provides a set of JAX-WS Services. Can call them all successfully using SOAP-Ui.

We have client code now on WebSphere which calls these same service endpoints, using JAX-WS Clients, deployed in a WebSphere JEE application. All work, except for 1 call, which works first time, then never again until WebSphere restart. The SOAP packet comes over the wire to us, but doesn't not appear to get unmarshalled correctly. Symptom for us is no response element (ReadAccountResponse) in the Java PoJo returned by the JAX-WS Client.

The SOAP response contains;

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <ns2:WSResponse xmlns:ns2="http://www.finxxx.com/wscomposer/operationtypes">
            <p:ReadAccountResponse xmlns:p="http://www.finxxx.com/wscomposer/ReadAccount">               
            </p:ReadAccountResponse>
        </ns2:WSResponse>
    </soapEnv:Body>
</soapenv:Envelope>

We have done a great deal of tracing on this, but can't get much further, I fear, without some help. We have detailed WebSphere Trace logs of the working case, and the failing case. The trace files are somewhat large :-) However, I paste below what I think are some relevent lines from each case. A guess here is that, first time in, WebSphere creates some kind of demarshaller for the SOAP response, and perhaps caches it. Second time around it seemt to try to retrieve it, but can't. I can paste more information if it's helpful to anyone.

The code (Client and Server) is deployed on WebSphere Application Server ND 8.5.5.

Extract from successful case, parsing of SOAP Response;

[12/02/21 10:13:07:329 GMT] 000000b6 BuilderUtil   1 org.apache.axis2.builder.BuilderUtil getCharSetEncoding char set encoding set from default =UTF-8
[12/02/21 10:13:07:329 GMT] 000000b6 StAXUtils     1 org.apache.axiom.om.util.StAXUtils createXMLStreamReader XMLStreamReader is org.apache.axiom.util.stax.dialect.DisallowDoctypeDeclStreamReaderWrapper
[12/02/21 10:13:07:330 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 7
[12/02/21 10:13:07:330 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState START_ELEMENT: 
[12/02/21 10:13:07:330 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://schemas.xmlsoap.org/soap/envelope/}Envelope
[12/02/21 10:13:07:330 GMT] 000000b6 StAXSOAPModel 1 org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder constructNode Starting to process SOAP 1.1 message
[12/02/21 10:13:07:330 GMT] 000000b6 StAXSOAPModel 1 org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder createOMElement Build the OMElement Envelope by the StaxSOAPModelBuilder
[12/02/21 10:13:07:330 GMT] 000000b6 AxisEngine    3 org.apache.axis2.engine.AxisEngine receive [MessageContext: logID=634c0f2975acb69481788a9d87a47ac49dc1ccb72dc039e5] receive:urn:uuid:212f0707-68f8-4ec3-9a30-dcde597aed6e
[12/02/21 10:13:07:331 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 1
[12/02/21 10:13:07:331 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState START_ELEMENT: 
[12/02/21 10:13:07:331 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://schemas.xmlsoap.org/soap/envelope/}Body
[12/02/21 10:13:07:331 GMT] 000000b6 StAXSOAPModel 1 org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder createOMElement Build the OMElement Body by the StaxSOAPModelBuilder
[12/02/21 10:13:07:331 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 1
.
. A lot of stuff here , Axis Engine transport related.
.
[12/02/21 10:13:07:352 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState START_ELEMENT: 
[12/02/21 10:13:07:352 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://www.fineos.com/wscomposer/operationtypes}WSResponse
[12/02/21 10:13:07:352 GMT] 000000b6 StAXSOAPModel 1 org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder createOMElement Build the OMElement WSResponse by the StaxSOAPModelBuilder
[12/02/21 10:13:07:352 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 1
[12/02/21 10:13:07:352 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState START_ELEMENT: 
[12/02/21 10:13:07:352 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://www.fineos.com/wscomposer/ReadAccount}ReadAccountResponse
[12/02/21 10:13:07:352 GMT] 000000b6 StAXSOAPModel 1 org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder createOMElement Build the OMElement ReadAccountResponse by the StaxSOAPModelBuilder
[12/02/21 10:13:07:353 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 1
[12/02/21 10:13:07:353 GMT] 000000b6 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState CHARACTERS: 
[12/02/21 10:13:07:353 GMT] 000000b6 XOPDecodingSt 1 org.apache.axiom.util.stax.xop.XOPDecodingStreamReader next Received Element Type is : 4

Extract from failing scenarios;

[23/02/21 11:12:09:594 GMT] 00000143 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState START_ELEMENT: 
[23/02/21 11:12:09:594 GMT] 00000143 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://www.fineos.com/wscomposer/operationtypes}WSResponse

[23/02/21 11:12:09:595 GMT] 00000143 StAXOMBuilder 1 org.apache.axiom.om.impl.builder.StAXOMBuilder createWithCustomBuilder Invoking CustomBuilder, org.apache.axis2.datasource.jaxb.JAXBCustomBuilder@7f5109cb, to the OMNode for {http://www.fineos.com/wscomposer/operationtypes}WSResponse
[23/02/21 11:12:09:595 GMT] 00000143 JAXBCustomBui 1 org.apache.axis2.datasource.jaxb.JAXBCustomBuilder create create namespace = http://www.fineos.com/wscomposer/operationtypes
[23/02/21 11:12:09:595 GMT] 00000143 JAXBCustomBui 1 org.apache.axis2.datasource.jaxb.JAXBCustomBuilder create   localPart = WSResponse
[23/02/21 11:12:09:595 GMT] 00000143 JAXBCustomBui 1 org.apache.axis2.datasource.jaxb.JAXBCustomBuilder create   reader = class org.apache.axiom.util.stax.xop.XOPDecodingStreamReader
.
. Seems to be looking for a cached unmarshaller, perhaps?
.
[23/02/21 11:12:09:599 GMT] 00000143 JAXBUtils     1 org.apache.axis2.jaxws.message.databinding.JAXBUtils getJAXBUnmarshaller Unmarshaller created [not in pool]
[23/02/21 11:12:09:599 GMT] 00000143 JAXBDSContext 1 org.apache.axis2.datasource.jaxb.JAXBDSContext unmarshal Adding JAXBAttachmentUnmarshaller to Unmarshaller
[23/02/21 11:12:09:599 GMT] 00000143 JAXBDSContext 1 org.apache.axis2.datasource.jaxb.JAXBDSContext unmarshalByElement Invoking unMarshalByElement
[23/02/21 11:12:09:599 GMT] 00000143 CompoundClass >  loadClass com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub this=com.ibm.ws.classloader.CompoundClassLoader@46aa37bd[PF][app:m6-cif-integration-ear_HAVEN_SIT] Entry
[23/02/21 11:12:09:600 GMT] 00000143 SinglePathCla 3   failed to read com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub from /opt/WebSphere8.5/AppServer/profiles/AppSvr01/installedApps/rhm6homvd1Cell01/m6-cif-integration-ear_HAVEN_SIT.ear/lib/m6-wscomposer-client-jar-0.0.1-SNAPSHOT.jar
[23/02/21 11:12:09:600 GMT] 00000143 JIT_StubPlugi >  defineStubClass: com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub Entry
[23/02/21 11:12:09:600 GMT] 00000143 JIT_StubPlugi <  defineStubClass: null Exit
[23/02/21 11:12:09:600 GMT] 00000143 CompoundClass <  loadClass com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub failed Exit
[23/02/21 11:12:09:604 GMT] 00000143 CompoundClass >  loadClass com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub this=com.ibm.ws.classloader.CompoundClassLoader@46aa37bd[PF][app:m6-cif-integration-ear_HAVEN_SIT] Entry
[23/02/21 11:12:09:604 GMT] 00000143 SinglePathCla 3   failed to read com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub from /opt/WebSphere8.5/AppServer/profiles/AppSvr01/installedApps/rhm6homvd1Cell01/m6-cif-integration-ear_HAVEN_SIT.ear/lib/m6-wscomposer-client-jar-0.0.1-SNAPSHOT.jar
[23/02/21 11:12:09:604 GMT] 00000143 JIT_StubPlugi >  defineStubClass: com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub Entry
[23/02/21 11:12:09:604 GMT] 00000143 JIT_StubPlugi <  defineStubClass: null Exit
[23/02/21 11:12:09:604 GMT] 00000143 CompoundClass <  loadClass com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse_JAXB_Deserialization_Stub failed Exit
[23/02/21 11:12:09:605 GMT] 00000143 JAXBUtils     1 org.apache.axis2.jaxws.message.databinding.JAXBUtils releaseJAXBUnmarshaller Unmarshaller placed back into pool
[23/02/21 11:12:09:606 GMT] 00000143 JAXBCustomBui 1 org.apache.axis2.datasource.jaxb.JAXBCustomBuilder create Successfully unmarshalled jaxb object com.fineos.wscomposer.operationtypes.contract.read.ReadAccountResponse@7f429fef
[23/02/21 11:12:09:606 GMT] 00000143 JAXBCustomBui 1 org.apache.axis2.datasource.jaxb.JAXBCustomBuilder create The JAXBDataSource is org.apache.axis2.datasource.jaxb.JAXBDataSource@c63fe9a9
[23/02/21 11:12:09:606 GMT] 00000143 OMSourcedElem 1 org.apache.axiom.om.impl.llom.OMSourcedElementImpl <init> OMSourcedElementImpl(String, OMNamespace, OMFactory, OMDataSource) ctor
[23/02/21 11:12:09:606 GMT] 00000143 OMSourcedElem 1 org.apache.axiom.om.impl.llom.OMSourcedElementImpl isComplete isComplete : isExpanded false
[23/02/21 11:12:09:606 GMT] 00000143 StAXOMBuilder 1 org.apache.axiom.om.impl.builder.StAXOMBuilder createWithCustomBuilder The CustomBuilder, org.apache.axis2.datasource.jaxb.JAXBCustomBuilder@7f5109cbsuccessfully constructed the OMNode for {http://www.fineos.com/wscomposer/operationtypes}WSResponse
[23/02/21 11:12:09:606 GMT] 00000143 StAXOMBuilder 1 org.apache.axiom.om.impl.builder.StAXOMBuilder createWithCustomBuilder The current state of the parser is: 
[23/02/21 11:12:09:606 GMT] 00000143 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState END_ELEMENT: 
[23/02/21 11:12:09:606 GMT] 00000143 StAXOMBuilder 3 org.apache.axiom.om.impl.builder.StAXOMBuilder logParserState   QName: {http://schemas.xmlsoap.org/soap/envelope/}Body

Answer 1

WebSphere has a jaxb impl that does some caching that might be amiss. You could try disabling this cache. Set the jvm argument -Dcom.ibm.xml.xlxp.jaxb.opti.level=0 to disable the cache and restart the server.