Pubsub message goes to dead letter immediately without any consuming attempt

Question

I'm currently facing problem where sometimes when a pubsub message is published, the message goes straight into the dead letter, without any attempt to deliver it to the corresponding subscriber. The delivery attempt is set to be 5. It does not even surpass the retention duration.

Reading google's documentation on Dead Letter policy, it is said that a message must reach the maximum delivery attempt before it goes to the dead letter, but in this case it goes straight to dead letter without any attempt.

My way to determine if a message is being delivered is to put log immediately as it reach the subscriber. And for this case, I don't see those logs. (Or is there's another method to trace message other than logging on subscriber?)

So far I don't find way to exactly reproduce this, as it only occurs randomly on some messages in production environment.

Answer 1

Pub/Sub moves messages to the dead letter topic after the configured number of delivery attempts. A delivery attempt does not mean the message got all the way to your subscriber necessarily. For example, if there is any network blip along the way that prevented delivery, this would count as a delivery attempt.

If you are using a push subscription, you can check the push_request_count metric to see if errors are being returned. These could be happening in a load balancer or other service between Pub/Sub and your actual subscriber.

If you are using a pull subscription, look at metrics like expired_ack_deadlines_count to see if messages are getting delivered to your subscriber client, but not acknowledged before the deadline. These expired deadlines would count as delivery attempts. If that metric shows a nonzero value, then messages are getting sent to your client and not getting processed in time, possibly due to flow control or due to the application code losing track of messages that were delivered to it. You should also look at the nack_requests metric to see if you are nacking messages, which would also count as a delivery attempt. If this metric is nonzero, it means you are nacking messages in your application.