CODEWITHSUNDEEP
jmsamqpspring-jmsazure-servicebus-queuesqpid
Ionut Iorga
Ionut Iorga

Reputation: 21

Azure Service Bus : Amqp Idle Timeout condition = amqp:link:detach-forced

The error I get:

2019-12-09 06:39:33.189 ERROR 107132 --- [http-nio-8082-exec-5] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.jms.IllegalStateException: The MessageProducer was closed due to an unrecoverable error.; nested exception is javax.jms.IllegalStateException: The MessageProducer was closed due to an unrecoverable error.] with root cause

javax.jms.JMSException: Idle link tracker, link qpid-jms:sender:ID:7300953e-f587-4ae3-b9fe-85b84e032554:1:101:1:order-update has been idle for 1800000ms TrackingId:801ab247-3f36-4470-8665-08846eb1c181_G24, SystemTracker:client-link34404815, Timestamp:2019-12-06T21:04:35 [condition = amqp:link:detach-forced]
    at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToException(AmqpSupport.java:164)
    at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToException(AmqpSupport.java:117)
    at org.apache.qpid.jms.provider.amqp.AmqpAbstractResource.processRemoteClose(AmqpAbstractResource.java:262)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:906)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:102)
    at org.apache.qpid.jms.provider.amqp.AmqpProvider$17.run(AmqpProvider.java:792)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

dependencies

compile group: 'com.microsoft.azure', name: 'azure-servicebus-spring-boot-starter', version: '0.2.0'
compile group: 'javax.jms', name: 'javax.jms-api', version: '2.0.1'
compile group: 'org.apache.qpid', name: 'qpid-jms-client', version: '0.28.0'
compile group: 'org.apache.camel', name: 'camel-jms', version: '2.24.1'
compile group: 'org.springframework.integration', name: 'spring-integration-jms', version: '5.0.4.RELEASE'

jmsConnectionFactory configuration:

<bean id="jmsConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory">
        <bean class="org.apache.qpid.jms.JmsConnectionFactory">
            <constructor-arg value="${azure.jms.url}" />
            <property name="username" value="${azure.jms.username}" />
            <property name="password" value="${azure.jms.password}" />
            <property name="clientID" value="AltaPay" />
            <property name="receiveLocalOnly" value="true" />
            <property name="localMessageExpiry" value="true" />
            <property name="populateJMSXUserID" value="true" />
        </bean>
    </property>
    <property name="exceptionListener">
        <bean class="com.lauraashley.microservice.altapay.callback.exception.CustomJMSExceptionListener" />
    </property>
    <property name="sessionCacheSize" value="10" />
    <property name="cacheConsumers" value="false" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
    <property name="connectionFactory" ref="jmsConnectionFactory" />
    <property name="cacheLevelName" value="CACHE_NONE" />
</bean>

CustomJMSExceptionListener

public class CustomJMSExceptionListener implements ExceptionListener {
    
    private static final Logger logger = getLogger(CustomJMSExceptionListener.class);

    @Override
    public void onException(JMSException exception) {
        // TODO Auto-generated method stub
        logger.error("--------------- Catched exception with CustomJMSExceptionListener ---------------");
        logger.error("Error code:"+exception.getErrorCode());
        logger.error("Msg:"+exception.getMessage());
        exception.printStackTrace();
        logger.error("---------------------------------------------------------------------------------");
    }
}

How I reproduce it

First: the CustomJMSExceptionListener is not used, isn't configured ok?

The application is a eccomerce app on OCC ( oracle cloud commerce ) platform that used java spring-boot services for the payment integration and flow.

This error occurs when an order exceeds the idle time then the connection with the Azure Service Bus fails and in order to reconnect I must restart the java app, and this is quite a big problem because no more orders can be processed. I read that CachingConnectionFactory has reconnectOnException which by default is true.

I don't really understand why this happens and what is the solution in order to fix it.

Upvotes: 2

Views: 6714

Answers (3)

user465033
user465033

Reputation: 71

This issue should be addressed for now. We have made a change on ServiceBus service side to not enforce the expiring of idle links aggressively for JMS customers that come through azure-servicebus-jms libraries (which is the dependent library used by spring-cloud-azure-starter-servicebus-jms, version 4.x ).

This is only available for premium messaging namespaces and using azure-servicebus-jms libraries or through the spring-cloud-azure-starter-servicebus-jms (version 4.x for now). There are still active quota enforcements on the number of active links that can be had for a namespace at any given point in time.

The real long term fix will likely need a fix involving Qpid JMS library, where a producer object on client is not immediately considered close on receiving a link close. Instead, when using the jmsproducer object, the qpid jms client library has to check if the underlying link is in closed state and if so re-create the underneath amqp link again for the same producer.

Upvotes: 0

Ismail
Ismail

Reputation: 1316

As answered above this is an expected behavior from Azure Service Bus, This issue is open in azure-spring-boot. As of now the workaround is to set the CachingConnectionFactory.cacheProducers value to False, so new Producer will be created for every session.

CachingConnectionFactory connectionFactory = (CachingConnectionFactory) jmsTemplate.getConnectionFactory();

connectionFactory.setCacheProducers(false);

Another possible way of doing this,

@Bean
    public ConnectionFactory jmsConnectionFactory(AzureServiceBusJMSProperties busJMSProperties){
        final String connectionString = busJMSProperties.getConnectionString();
        final String clientId = busJMSProperties.getTopicClientId();
        final int idleTimeout = busJMSProperties.getIdleTimeout();

        final ServiceBusKey serviceBusKey = ConnectionStringResolver.getServiceBusKey(connectionString);

        final String remoteUri = String.format("amqps://%s?amqp.idleTimeout=%d&amqp.traceFrames=true",
                serviceBusKey.getHost(), idleTimeout);

        final JmsConnectionFactory jmsConnectionFactory =
                new JmsConnectionFactory(
                        serviceBusKey.getSharedAccessKeyName(),
                        serviceBusKey.getSharedAccessKey(),
                        remoteUri
                );
        jmsConnectionFactory.setClientID(clientId);

        CachingConnectionFactory cachingConnectionFactory =
                new CachingConnectionFactory(jmsConnectionFactory);
                // set cache producers to FALSE here
        cachingConnectionFactory.setCacheProducers(false);

        return cachingConnectionFactory;
    }

Upvotes: 2

Tim Bish
Tim Bish

Reputation: 18356

The exception is indicating that Azure has closed the producer because it was idle for to long, meaning it hadn't sent a message within the timeout (some documentation here). You might be able to work around this when using the CachingConnectionFactory by configuring the cache producers option to false so that producers are created on demand but I'm not entirely sure on that as I don't have any way to test it.

This isn't a Qpid JMS client level bug but rather the behaviour of Azure kicking in where after I think it's ten minutes of no activity on a link it will forcibly close the link. In a non-spring based application you'd have to account for this by catching the JMSException on send either attempting to create a new producer and sending again or by tearing down the whole connection and starting over. Your reaction somewhat depends on foreknowledge that you are using Azure and knowing that this can happen.

Upvotes: 4

Related Questions