CODEWITHSUNDEEP
activemq-artemis
skin27
skin27

Reputation: 426

org.apache.activemq.artemis.core.client : AMQ214013: Failed to decode packet

I have a cluster of two ActiveMQ Artemis 2.18.0 brokers. The "server2" broker has the following in its broker.xml:

<connectors>
       <connector name="netty-connector">tcp://server1:61616?sslEnabled=true;trustStorePath=/data/assimbly/broker/.assimbly/security/truststore-01.jks;truststorePassword=secret</connector>
</connectors>
        
<acceptors>
     <acceptor name="netty-acceptor">tcp://0.0.0.0:61616?protocols=CORE,AMQP,STOMP,HORNETQ,OPENWIRE;sslEnabled=true;keyStorePath=/data/assimbly/broker/.assimbly/security/keystore-01.jks;keyStorePassword=secret</acceptor>
         
     <acceptor name="amqp">tcp://0.0.0.0:5672?protocols=CORE,AMQP;sslEnabled=true;keyStorePath=/data/assimbly/broker/.assimbly/security/keystore-01.jks;keyStorePassword=secret</acceptor>
</acceptors>

    <cluster-connections>
       <cluster-connection name="test-cluster">
          <connector-ref>netty-connector</connector-ref>
          <retry-interval>1000</retry-interval>
          <use-duplicate-detection>false</use-duplicate-detection>
          <message-load-balancing>ON_DEMAND</message-load-balancing>
          <max-hops>1</max-hops>
          <discovery-group-ref discovery-group-name="test-discovery-group"/>
       </cluster-connection>
    </cluster-connections>

The broker starts normally without SSL enabled. However, when SSL is enabled in broker.xml the broker starts (on server2) with the following error:

org.apache.activemq.artemis.core.client : AMQ212037: Connection failure to server1/10.96.1.6:61616 has been detected: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38 [code=GENERIC_EXCEPTION]
org.apache.activemq.artemis.core.client : AMQ214013: Failed to decode packet
java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38

Currently, there are no messages on the broker (fresh start). I can't pinpoint where the "length=1970500723" is coming from.

What could be the issue?

Stack trace:

2021-10-05 12:37:50,496 WARN  [Thread-3 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f8d7d07)] client: AMQ212037: Connection failure to lnx00057.svb.org/10.96.1.6:61616 has been detected: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38 [code=GENERIC_EXCEPTION]
2021-10-05 12:37:50,508 ERROR [Thread-10 (ActiveMQ-client-netty-threads)] client: AMQ214013: Failed to decode packet
java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38
        at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:185)
        at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:173)
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readStringInternal(ChannelBufferWrapper.java:113)
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableString(ChannelBufferWrapper.java:88)
        at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V3.decodeRest(ClusterTopologyChangeMessage_V3.java:81)
        at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V4.decodeRest(ClusterTopologyChangeMessage_V4.java:56)
        at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371)
        at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277)
        at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388)
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263)
        at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2021-10-05 12:37:50,508 ERROR [Thread-10 (ActiveMQ-client-netty-threads)] client: AMQ214031: Failed to decode buffer, disconnect immediately.
java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:401)
        at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263)
        at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38
        at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:185)
        at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:173)
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readStringInternal(ChannelBufferWrapper.java:113)
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableString(ChannelBufferWrapper.java:88)
        at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V3.decodeRest(ClusterTopologyChangeMessage_V3.java:81)
        at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V4.decodeRest(ClusterTopologyChangeMessage_V4.java:56)
        at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:371)
        at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277)
        at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149)
        at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:388)
        ... 29 common frames omitted
2021-10-05 12:37:50,508 WARN  [Thread-1 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f8d7d07)] client: AMQ212037: Connection failure to lnx00057.svb.org/10.96.1.6:61616 has been detected: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1970500723 is greater than readableBytes=38 [code=GENERIC_EXCEPTION]
2021-10-05 12:37:50,530 WARN  [Thread-2 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f8d7d07)] client: AMQ212025: did not connect the cluster connection to other nodes
org.apache.activemq.artemis.api.core.ActiveMQConnectionTimedOutException: AMQ219013: Timed out waiting to receive cluster topology. Group:org.apache.activemq.artemis.core.cluster.DiscoveryGroup@2f9129bc
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:743)
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:486)
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
        at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2021-10-05 12:37:50,531 WARN  [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@1f8d7d07)] server: AMQ224091: Bridge ClusterConnectionBridge@5ba37c27 [name=$.artemis.internal.sf.test-cluster.918859b6-25f5-11ec-8bff-001dd8c12fe4, queue=QueueImpl[name=$.artemis.internal.sf.test-cluster.918859b6-25f5-11ec-8bff-001dd8c12fe4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=02], temp=false]@55648923 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@5ba37c27 [name=$.artemis.internal.sf.test-cluster.918859b6-25f5-11ec-8bff-001dd8c12fe4, queue=QueueImpl[name=$.artemis.internal.sf.test-cluster.918859b6-25f5-11ec-8bff-001dd8c12fe4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=02], temp=false]@55648923 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&sslEnabled=true&host=lnx00057-svb-org&trustStorePath=/data/assimbly/broker/-assimbly/security/truststore-01-jks], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1709963701[nodeUUID=98f793bf-25f5-11ec-9e30-001dd8c12fe5, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?trustStorePassword=****&port=61616&sslEnabled=true&host=lnx00058-svb-org&trustStorePath=/data/assimbly/broker/-assimbly/security/truststore-01-jks, address=, server=ActiveMQServerImpl::name=02])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&sslEnabled=true&host=lnx00057-svb-org&trustStorePath=/data/assimbly/broker/-assimbly/security/truststore-01-jks], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying

The last log line indicates that the discoveryGroupConfiguration=null. I'm not sure if this is directly related or a separated issue. Here is the configuration of these groups:

    <broadcast-groups>
       <broadcast-group name="test-broadcast-group">
        <group-address>${udp-address:231.7.7.7}</group-address>
        <group-port>9876</group-port>
        <broadcast-period>2000</broadcast-period>
        <connector-ref>netty-connector</connector-ref>
       </broadcast-group>
    </broadcast-groups>

      <discovery-groups>
         <discovery-group name="test-discovery-group">
            <group-address>${udp-address:231.7.7.7}</group-address>
            <group-port>9876</group-port>
            <refresh-timeout>10000</refresh-timeout>
         </discovery-group>
      </discovery-groups>

I tried changing the group-address of my discovery-group and broadcast-group to use 231.6.6.6 instead of 231.7.7.7 on both nodes, but the failure still occurred.

Upvotes: 0

Views: 3020

Answers (2)

Domenico Francesco Bruscino
Domenico Francesco Bruscino

Reputation: 2309

To fix this error you should replace truststorePassword with trustStorePassword because the check for allowed connector parameters is case sensitive.

This issue is caused by the not allowed connector parameters because they are handled as extra parameters and there is an error on encoding and decoding TransportConfiguration ExtraParams, see ARTEMIS-3363 for further details.

Upvotes: 1

skin27
skin27

Reputation: 426

This error occurs because on connecting to the other node, the url in the connector is too long for the simpleString:

<connectors>
       <connector name="netty-connector">tcp://server1:61616?sslEnabled=true;trustStorePath=/data/assimbly/broker/.assimbly/security/truststore-01.jks;truststorePassword=secret</connector>
</connectors>

The error message gives some information:

Error reading in simpleString, length=1970500723 is greater than readableBytes=38

The length=1970500723 is not the length of characters or bytes. It's part of the URL. When decoding it:

  Decimal to Hexadecimal (1970500723 --> 75737473)
  Hexadecimal to ASCII (75737473 --> usts)

The 'usts' relates to 'trustStorePath' in the connector. This URL parameter is a valid way for configuring the transport. On testing the error didn't occur when using the standalone Artemis distribution, but it did occur when using an embedded broker.

The solution was to change the URL to:

<connectors>
       <connector name="netty-connector">tcp://server1:61616?sslEnabled=true</connector>
</connectors>

And then start the java application with the following system properties:

java -Djavax.net.ssl.trustStore=path/to/truststore.jks  -Dorg.apache.activemq.ssl.trustStore=secret -jar MyApp.jar

Upvotes: 0

Related Questions