CODEWITHSUNDEEP
azureapachejmsazureservicebus
fatherazrael
fatherazrael

Reputation: 5987

Azure Service Bus | AMQPS | Unstable Every month | Unknown error from remote peer

It happens yesterday too and message starts from following:

2024-09-19 23:36:18.880  INFO 1776090 --- [569306cb73d7:39] org.apache.qpid.jms.JmsSession           : A JMS MessageConsumer has been closed: JmsConsumerInfo: { ID:45157fc8-58d2-44d9-b552-569306cb73d7:39:1:1, destination = s-ataccama-generalsupplier }
2024-09-19 23:36:18.880  WARN 1776090 --- [569306cb73d7:39] o.a.qpid.jms.util.QpidJMSThreadFactory   : Thread: QpidJMS Connection Executor: ID:45157fc8-58d2-44d9-b552-569306cb73d7:39 failed due to an uncaught exception: null

After these two lines there is following stack trace:

   2024-09-19 23:36:18.881  INFO 1776090 --- [plierContact-36] o.s.j.c.CachingConnectionFactory         : Encountered a JMSException - resetting the underlying JMS Connection
    
    javax.jms.JMSException: Unknown error from remote peer
            at org.apache.qpid.jms.provider.ProviderException.toJMSException(ProviderException.java:34) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:80) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:112) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:915) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:899) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.JmsMessageConsumer.performPullIfRequired(JmsMessageConsumer.java:732) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.JmsMessageConsumer.dequeue(JmsMessageConsumer.java:304) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.apache.qpid.jms.JmsMessageConsumer.receive(JmsMessageConsumer.java:213) ~[qpid-jms-client-0.53.0.jar!/:na]
            at org.springframework.jms.support.destination.JmsDestinationAccessor.receiveFromConsumer(JmsDestinationAccessor.java:132) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:431) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:316) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:270) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120) ~[spring-jms-5.3.23.jar!/:5.3.23]
            at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
    Caused by: org.apache.qpid.jms.provider.ProviderException: Unknown error from remote peer
        at org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToNonFatalException(AmqpSupport.java:184) ~[qpid-jms-client-0.53.0.jar!/:na]
        at org.apache.qpid.jms.provider.amqp.AmqpAbstractResource.processRemoteClose(AmqpAbstractResource.java:269) ~[qpid-jms-client-0.53.0.jar!/:na]
        at org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:985) ~[qpid-jms-client-0.53.0.jar!/:na]
        at org.apache.qpid.jms.provider.amqp.AmqpProvider.onData(AmqpProvider.java:871) ~[qpid-jms-client-0.53.0.jar!/:na]
        at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:563) ~[qpid-jms-client-0.53.0.jar!/:na]
        at org.apache.qpid.jms.transports.netty.NettyTcpTransport$NettyTcpTransportHandler.channelRead0(NettyTcpTransport.java:556) ~[qpid-jms-client-0.53.0.jar!/:na]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) ~[netty-handler-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519) ~[netty-codec-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458) ~[netty-codec-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.82.Final.jar!/:4.1.82.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.82.Final.jar!/:4.1.82.Final]

After this i get following logs:

2024-09-19 23:36:18.992  WARN 1776090 --- [plierContact-36] o.s.j.l.DefaultMessageListenerContainer  : Setup of JMS message listener invoker failed for destination 's-ataccama-generalsupplier' - trying to recover. Cause: Unknown error from remote peer
2024-09-19 23:36:19.278  INFO 1776090 --- [windows.net:-1]] org.apache.qpid.jms.JmsConnection        : Connection ID:8405d717-54fd-49e9-ad40-534f6f3c2548:40 connected to server: amqps://tietoator.servicebus.windows.net
2024-09-19 23:36:39.714  WARN 1776090 --- [windows.net:-1]] o.a.q.j.p.a.b.AmqpResourceBuilder        : Open of resource:(JmsConsumerInfo: { ID:8405d717-54fd-49e9-ad40-534f6f3c2548:40:1:1, destination = s-ataccama-generalsupplier }) failed: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:bc369da39d964bc39c6bf63281bd3182_G16, SystemTracker:gateway10, Timestamp:2024-09-19T21:36:39 [condition = amqp:internal-error]

and also

Caused by: org.apache.qpid.jms.provider.ProviderException: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:bc369da39d964bc39c6bf63281bd3182_G16, SystemTracker:gateway10, Timestamp:2024-09-19T21:36:39 [condition = amqp:internal-error] 2024-09-19 23:36:44.866  WARN 1776090 --- [plierContact-37] o.s.j.l.DefaultMessageListenerContainer  : Setup of JMS message listener invoker failed for destination 's-ataccama-generalsupplier' - trying to recover. Cause: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:bc369da39d964bc39c6bf63281bd3182_G16, SystemTracker:gateway10, Timestamp:2024-09-19T21:36:39 [condition = amqp:internal-error]

Using Standard Service Bus with Spring Boot JMS:

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.7.4</version>
            <type>pom</type>
            <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.azure.spring</groupId>
                <artifactId>spring-cloud-azure-dependencies</artifactId>
                <version>4.3.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

This error comes twice a month and some poor people like me are searching this for an year :-) https://github.com/Azure/azure-service-bus-java/issues/424

Upvotes: 0

Views: 115

Answers (1)

Zooming Rocket
Zooming Rocket

Reputation: 49

We have been encountering similar mysterious consumer connectivity issues with Azure Servicebus while connecting using Apache Qpid JMS Client. We incorporated failover with auto connection retry configuration as a potential mitigation for improved resiliency!

Configuration for Qpid Properties at ConnectionFactory

connectionfactory.MYCONNFACTORY = failover:(amqps://namespace.servicebus.windows.net:5671,amqps://namespace.servicebus.windows.net:5671)?failover.initialReconnectDelay=10000&failover.reconnectDelay=5000&failover.maxReconnectDelay=600000&failover.useReconnectBackOff=true&failover.warnAfterReconnectAttempts=1&jms.username=xxxxxx&jms.password=xxxxxxx&amqp.idleTimeout=120000&amqp.traceFrames=true&jms.prefetchPolicy.queuePrefetch=10&transport.tcpKeepAlive=true&transport.enabledProtocols=TLSv1.2

queue.[jndiName] = [physicalName]

Property Explanation available at Qpid Reference: https://qpid.apache.org/releases/qpid-jms-1.12.1/docs/index.html

Overall we suspect Azure side issue between Basic v/s Premium Tier possibly causing intermittent failures due to JMS 1.1 vs JMS 2.0 support differences enforced through Servicebus Product Tiers

Azure Service Bus Guide for JMS 1.1 with Qpid - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-java-how-to-use-jms-api-amqp

Azure Service Bus (Premium Tier Only) Guide for JMS 2.0 + Azure Servicebus JMS Library - https://learn.microsoft.com/en-us/azure/service-bus-messaging/how-to-use-java-message-service-20

Upvotes: 0

Related Questions