CODEWITHSUNDEEP
javaspringspring-bootrabbitmqspring-integration
vipulk10
vipulk10

Reputation: 119

Spring boot 2.0 service : messages delivered from Rabbit mq but not reaching java code

We are using a high powered rabbit mq consumer built on top of spring boot 2.0 and using spring integration framework. Prefetch count is set to 150 and no of concurrent consumers are 4 . Everything was working fine till one day some of messages remain unacked in rmq . When we restart our java service the unacked messages get processed but the problem creeps in over certain time again. Can anyone help with this strange problem . Right now we have set up a cron job to restart the service after 1 hour so that no messages remain unacked.

PS spring boot version 2.0 rabbit mq version : 3.7.7

EDIT

The problem has returned in another service. We have upgraded our spring boot service to 2.1 version and latest spring-amqp version. The messages keep on increasing in unacked state with no logs coming in the service . I am attaching threaddump of our service

Threaddump : https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMTIvMTcvLS1hcGktZjA3MmQ1NDUtZDRlYy00ZGQxLThkZjktN2M2YzAyYjM0ZTMxZTUwZWUyYTktM2EyMC00ZDhkLThlZjctM2NmZjVlZmVlNTdkLnR4dC0t&

PS on analysing some messages I believe that the problem lies in spring integration channels somehow . I had enabled debug logs in our service and found a particular log which caught my attention : 

2019-12-17 ; 07:33:35.111 ;  WARN ; 1 --- [       Thread-3] ; o.s.a.r.l.SimpleMessageListenerContainer ;  ; Closing channel for unresponsive consumer: Consumer@7bc9e6ab: tags=[[]], channel=Cached Rabbit Channel: null, conn: Proxy@40620d8e Shared Rabbit Connection: SimpleConnection@4c060c8f [delegate=amqp://[email protected]:5672/, localPort= 45896], acknowledgeMode=AUTO local queue size=0

Can some one throw some light on this

Upvotes: 0

Views: 1404

Answers (1)

Gary Russell
Gary Russell

Reputation: 174749

There are, generally, two possible causes for this - the listener threads are "stuck" in user code (this has been the problem in the majority of cases) or the rabbitmq client library has autoRecoveryEnabled=true; in general Spring AMQP sets it to false but there are some cases when it can be set to true and there were some corner cases where consumers were left "dangling". Spring AMQP doesn't need it because it has its own connection recovery mechanisms.

For the first case, take a thread dump to see what the threads are doing.

For the second case, upgrade to the latest spring amqp (2.0.8 - boot 2.0.6) or 2.1.0 (which will be the version in boot 2.1).

Upvotes: 1

Related Questions