CODEWITHSUNDEEP
publishpublish-subscribepublishinggoogle-cloud-pubsub
user3451476
user3451476

Reputation: 307

Set Expiry of GoogleCloudPubsub Message

Is there a way to set the expiry of a Google Pubsub message(Bad request) that I publish, so that it does not retry indefinitely on failure ?

I cannot configure this with retry because I want valid Error messages to retry indefinitely

Upvotes: 3

Views: 1354

Answers (1)

Takashi Matsuo
Takashi Matsuo

Reputation: 3436

There is only a global level 7-days expiration. However, you can add timestamp as an attribute and check the timestamp at the beginning of your pipeline, then throw away if conditions are met.

That said, we don't recommend that you only use timestamp for determining whether or not you throw away the message because if you have a big backlog and the consumer can not catch up, it is possible that valid messages are thrown away even if it's the first time being processed.

Here is another idea. When you publish the message, you get the message IDs in the API response, which you can later use in order to identify individual messages. In your pipeline, you can increment the retry counts on each message IDs, so that you know how many times the retry occurred for a particular message. Then you can throw away messages which are retried more than N times. This strategy is more reliable in my opinion. These retry count is not a critical thing, so you may be able to store them just in memory.

Upvotes: 3

Related Questions