CODEWITHSUNDEEP
spring-jmsactivemq-artemis
dim5b
dim5b

Reputation: 29

Understanding ActiveMQ Artemis AMQ222061 & AMQ222107 warnings

Currently I am receiving very low traffic to my application 1-2 req/sec which are sent to an JMS queue. However in the artemis.log I am seeing the following WARN messages.

2021-06-23 10:51:49,433 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session d798a4d3-d3f7-11eb-86d7-005056b536f0
2021-06-23 10:51:49,434 WARN  [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session d798a4d3-d3f7-11eb-86d7-005056b536f0
2021-06-23 10:51:49,434 WARN  [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session d798cbe4-d3f7-11eb-86d7-005056b536f0
2021-06-23 10:51:49,435 WARN  [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session d798cbe4-d3f7-11eb-86d7-005056b536f0

The Artemis journal is backed by NFS, and all messages are persisted. I am using Spring's JmsTemplate with a JmsPoolConnectionFactory.

@Bean
public ConnectionFactory jmsConnectionFactoryOnline() {
    ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(brokerUrl,username,password);
    connectionFactory.setCallTimeout(3000);
    return connectionFactory;
}

@Bean
@Primary
public JmsPoolConnectionFactory pooledConnectionFactory() {
    JmsPoolConnectionFactory poolingFactory = new JmsPoolConnectionFactory();
    poolingFactory.setConnectionFactory(jmsConnectionFactoryOnline());
    poolingFactory.setMaxConnections(8);
    poolingFactory.setMaxSessionsPerConnection(20);
    poolingFactory.setConnectionIdleTimeout(0);
    return poolingFactory;
}

Any ideas why there so many disconnects (they repeat every ~15 minutes) and how to debug it?

At client level I get the below warning but 3 seconds seems to much since the broker is up and accepting other messages:

23-06-2021 09:33:15 [Thread-24 (ActiveMQ-client-global-threads)] [WARN ] org.apache.activemq.artemis.core.client - AMQ212037: Connection failure to XXXXXX:61616 has been detected: AMQ219014: Timed out after waiting 3,000 ms for response when sending packet 71 [code=CONNECTION_TIMEDOUT]

EDIT

I have taken a thread dump every 3 sec. Check here for relative information.

Monitoring the push to queue latency I can clearly see that with 1 req/sec I have spikes of latency whereas with 20 req/sec push to queue latencies seem normal as observed in the image below.

Push to Queue latency

Also from tcpdump I can see that there is a pause between client and broker. There seems to be no traffic exchanged between them. Therefore as soon as the callTimeout period has elapsed the client closes the connection.

tcpdump

Upvotes: 1

Views: 4271

Answers (1)

Justin Bertram
Justin Bertram

Reputation: 34988

It's impossible to tell why there are so many disconnections with the information you've provided, but here are some debugging ideas:

  • Get thread dumps from the broker to see if it's busy during the time around the disconnections.
  • Turn on GC logging to ensure that long pauses are not causing the broker to be unresponsive.
  • Turn on TRACE logging for org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl. See the ActiveMQ Artemis documentation for details on how to configure that. This will let you see the incoming PING packets from clients or the absence thereof. When the broker doesn't receive pings within the configured connection TTL then it will close the client's connection.
  • Monitor network resources for problems like delays, dropped packets, disconnections, etc.

Upvotes: 1

Related Questions