CODEWITHSUNDEEP
javaspringactivemq-classicweblogic-10.xspring-batch
Andrew Regan
Andrew Regan

Reputation: 5113

JMS reader within Spring Batch causes NoClassDefFoundError

We're using Spring Batch 2.1.9 with Spring 3.1.1, and have set up a Job which does the following:

This is all deployed as a WAR, running under Weblogic 10.3.x. Now, we've had this design working OK elsewhere, but on this particular deployment, if a new message is sent to the ItemReader after a good deal of inactivity, the following occurs:

09:20:28,155 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat operation about to start at count=2
10:39:38,733 - WARN  (org.springframework.jms.connection.CachingConnectionFactory) - Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Unexpected error occured: java.lang.NoClassDefFoundError: org/apache/activemq/command/ActiveMQObjectMessage
    at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
    at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1934)
    at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1951)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
    at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:268)
    at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:213)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unexpected error occured: java.lang.NoClassDefFoundError: org/apache/activemq/command/ActiveMQObjectMessage
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:211)
    ... 1 more
Caused by: java.lang.NoClassDefFoundError: org/apache/activemq/command/ActiveMQObjectMessage
    at org.apache.activemq.openwire.v9.ActiveMQObjectMessageMarshaller.createObject(ActiveMQObjectMessageMarshaller.java:55)
    at org.apache.activemq.openwire.OpenWireFormat.tightUnmarshalNestedObject(OpenWireFormat.java:446)
    at org.apache.activemq.openwire.v9.BaseDataStreamMarshaller.tightUnmarsalNestedObject(BaseDataStreamMarshaller.java:125)
    at org.apache.activemq.openwire.v9.MessageDispatchMarshaller.tightUnmarshal(MessageDispatchMarshaller.java:71)
    at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:371)
    at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
    at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:229)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:221)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
    ... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.activemq.command.ActiveMQObjectMessage
    at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:297)
    at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
    at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
    at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:43)
    ... 10 more

Which seems to then be followed by a Session Closed (we are using Spring's CachingConnectionFactory, FWIW)

10:39:38,780 - DEBUG (org.springframework.batch.core.step.item.FaultTolerantChunkProvider) - Skipping failed input
org.springframework.jms.IllegalStateException: The Session is closed; nested exception is javax.jms.IllegalStateException: The Session is closed
    at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:279)
    at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
    at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
    at org.springframework.jms.core.JmsTemplate.receiveSelected(JmsTemplate.java:695)
    at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:677)
    at org.springframework.jms.core.JmsTemplate.receive(JmsTemplate.java:669)
    at org.springframework.jms.core.JmsTemplate.receiveAndConvert(JmsTemplate.java:787)
    at org.springframework.batch.item.jms.JmsItemReader.read(JmsItemReader.java:84)
    at our.code.spring.batch.AquilaJmsItemReader.read(AquilaJmsItemReader.java:61)
    at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:90)
    at org.springframework.batch.core.step.item.FaultTolerantChunkProvider.read(FaultTolerantChunkProvider.java:87)
    at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:110)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
    at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:105)
    at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:68)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
    at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.IllegalStateException: The Session is closed
    at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:731)
    at org.apache.activemq.ActiveMQSession.getTransacted(ActiveMQSession.java:521)
    at sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:348)
    at $Proxy1006.getTransacted(Unknown Source)
    at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:742)
    at org.springframework.jms.core.JmsTemplate.doReceive(JmsTemplate.java:722)
    at org.springframework.jms.core.JmsTemplate$9.doInJms(JmsTemplate.java:697)
    at org.springframework.jms.core.JmsTemplate$9.doInJms(JmsTemplate.java:1)
    at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
    ... 32 more

After about 20 or so more NoClassDefFoundError traces, the read operation finishes:

10:39:41,624 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat is complete according to policy and result value.
10:39:41,624 - DEBUG (org.springframework.batch.core.step.item.ChunkOrientedTasklet) - Inputs not busy, ended: false
10:39:41,624 - DEBUG (org.springframework.batch.core.step.tasklet.TaskletStep) - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=100, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
10:39:41,639 - DEBUG (org.springframework.batch.core.step.tasklet.TaskletStep) - Saving step execution before commit: StepExecution: id=1, version=5, name=fetchEmailsStep, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=500, writeSkipCount=0, processSkipCount=0, commitCount=5, rollbackCount=0, exitDescription=
10:39:41,639 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat operation about to start at count=6
10:39:41,639 - DEBUG (org.springframework.batch.core.scope.context.StepContextRepeatCallback) - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext@306897b6
10:39:41,639 - DEBUG (org.springframework.batch.core.scope.context.StepContextRepeatCallback) - Chunk execution starting: queue size=0
10:39:41,639 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Starting repeat context.
10:39:41,639 - DEBUG (org.springframework.batch.repeat.support.RepeatTemplate) - Repeat operation about to start at count=1

But the net result is that our message does not get passed to the ItemWriter

Can anyone explain why we might suddenly get NoClassDefFoundError errors?

I'd really like to avoid having to start changing the way we deploy just to work around a Weblogic classloader problem, e.g. having to move the ActiveMQ JARs to \server\lib, etc.

Our Spring configuration:

<bean id="jmsEmailFetcher" class="our.code.batch.AquilaJmsItemReader">
    <property name="jmsTemplate" ref="batchEmailJmsTemplate" />
</bean>

<bean id="batchEmailJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <property name="connectionFactory" ref="springBatchJmsCachingConnF"/>
    <property name="defaultDestination" ref="springBatchIncomingEmailQueue" />
    <!-- Do *not* need to set a timeout: <property name="receiveTimeout" value="1800000" /> -->
</bean>

<bean id="springBatchJmsCachingConnF" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory" ref="srcConnectionFactory"/>
    <property name="reconnectOnException" value="true"/>
    <!-- Leave "sessionCacheSize" as default (1) until we have a better answer -->
</bean>

<jee:jndi-lookup id="srcConnectionFactory" jndi-name="${jndi.jms.connfactory}">
    <jee:environment>
        java.naming.factory.initial = <snip>
        java.naming.provider.url = <snip>
    </jee:environment>
</jee:jndi-lookup>

Upvotes: 2

Views: 3243

Answers (1)

Andrew Regan
Andrew Regan

Reputation: 5113

If anyone's interested, almost certainly the cause of this is something like what follows:

  • App redeploys on Weblogic at regular intervals, causing:
  • Classes to get unloaded unexpectedly.
  • Long-running polling processes detect the missing class(es), and throw exceptions.
  • Other processes listening for the end of tasks kick into action, and themselves fail to work properly.

Upvotes: 2

Related Questions