CODEWITHSUNDEEP
google-cloud-platformpublish-subscribe
Bernard Wiesner
Bernard Wiesner

Reputation: 1435

How often do duplicates occur if I do not set "exactly once delivery" with pubsub?

I have noticed that using "Exactly once delivery" affects performance when using pull and acknowledge. Pull and acknowledge messages take up to 5 times longer ~0.2s. If I disable the "Exactly once delivery" response is much fast, under 0.05s for both pull and acknowledge. I tested using curl and php with similar results (reusing existing connection).

I am concerned what is the consequence of disabling this feature. How often do duplicates occur if this feature is disabled? Are there ways to avoid duplicates without enabling this feature?

For example, if I have an acknowledge deadline of 60 seconds, I pull a message then pull again after 10 seconds, could I get the same message again? Its unclear from the docs how often duplicates will occur and under what circumstances they will occur if this option is disabled.

Upvotes: 0

Views: 496

Answers (2)

André Figueira
André Figueira

Reputation: 6726

In the system I use, duplicates were happening often enough to cause us massive problems.

Upvotes: 1

somethingsomething
somethingsomething

Reputation: 2189

How often do duplicates occur if this feature is disabled?

Not super often in my experience, but this doesn't matter, your system needs to be able to handle them one way or another, because it will happen.

Are there ways to avoid duplicates without enabling this feature?

On googles side? No, otherwise what would be the point of the option. The user should either de-duplicate it with the messageID, by only processing each id once, or make sure that whatever operation you perform is idempotent. Or you don't bother, hope it doesn't happen often and live with the consequences (be it by crashing, having corruption somewhere that you may or may not fix,...).

Its unclear from the docs how often duplicates will occur and under what circumstances they will occur if this option is disabled.

Pub/sub is a complex highly scaling distributed system, duplicated messages are not an intended feature on a fixed schedule, they are a necessary evil if you want high performance. Nobody can predict when they will happen, only that they can occur.

Upvotes: 1

Related Questions