CODEWITHSUNDEEP
google-cloud-pubsub
Lina
Lina

Reputation: 73

PubSub: StatusRuntimeException: UNAVAILABLE from StreamingSubscriberConnection

We are running our Spring Boot app in a GCP with Kubernetes. We are using spring-cloud-gcp-starter-pubsub 1.1.0.RC1 -> google-cloud-pubsub:1.54.0. Lately we started to get exceptions:

logger_name com.google.cloud.pubsub.v1.StreamingSubscriberConnection

method onFailure

severity WARN

stack_tracecom.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE:

with exception messages like: 

 - Network closed for unknown reason io exception 
 - HTTP/2 error code:
 - NO_ERROR Received Goaway max_age 
 - Authentication backend unavailable.
 - The service was unable to fulfill your request. Please try again.
   [code=8a75] 502:Bad Gateway

The exceptions most often come in clusters: 

Feb 04 11:29:00.615
Feb 04 11:29:00.479
Feb 04 10:35:48.256
Feb 04 10:35:32.024
Feb 04 10:35:03.760
Feb 04 10:34:52.094
Feb 04 07:36:31.430
Feb 04 07:06:17.025
Feb 04 06:42:13.529
Feb 04 04:32:50.265
Feb 04 04:32:49.845
Feb 04 04:32:49.746
Feb 04 02:57:36.678
Feb 04 02:57:35.700

We get about 10 of these exceptions each day and can not find any relation to something happening in the system like deploys, heavy load etc.

My questions are:

Upvotes: 0

Views: 1705

Answers (1)

Daniel Collins
Daniel Collins

Reputation: 812

If you did not yet ack the messages, Cloud Pub/Sub will continue trying to deliver them until you do ack them or they expire based on your settings.

Without knowing more about your code, its difficult to give you any advice on how to proceed. Long standing grpc channels (such as the streaming pull channel used for subscribers) can be broken by transient network errors. I'd suggest you proceed by filing a bug against the corresponding client libraries you're using to see if they can resolve this issue or push this upstream to the Cloud PubSub client libraries which they are (presumably) using.

Upvotes: 1

Related Questions