CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-pubsub
skjagini
skjagini

Reputation: 3217

GCP Pub/Sub, can you replay old messages on a new Subscription if there is already an active Subscription

In GCP Pub/Sub I have a topic and a Subscription1 and have have started publishing messages.

Can I add another subscription, Subscription2 and replay old messages that had been published before the Subscription2 was created? Would it allow it? (Kafka allows it)

At what point would I lose access to messages (within the retention period)?, deleting all the Subscriptions? Would keeping at least one active subscription allow PubSub to add new Subscriptions and replay old messages?

Can I also increase the retention period beyond 7 days on the topic?

Edit: The messages will be persisted to DB, but I am more interested in a pub/sub architecture

Upvotes: 4

Views: 4048

Answers (3)

Jakub Kukul
Jakub Kukul

Reputation: 14572

Replaying old topic messages to a new subscription is possible since mid-2021 (see the announcement for more details).

To be able to do this, message retention must be enabled for the topic (maximum configurable retention is 31 days).

To do this, using gcloud CLI, you'd first create a new subscription as usual:

gcloud pubsub subscriptions create test-subscription --topic test-topic

and then "replay" old messages retained on the topic using the seek command:

gcloud pubsub subscriptions seek test-subscription \
  --time="$(date --date='10 days ago')"

In this example messages from 10 days in the past would become available in the subscription. Please note that the date subcommand won't work on macOS but you could install and use gdate instead (brew install gdate).

Upvotes: 3

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17261

It would not be good to rely on being able to replay messages in Subscription2 based on the fact that Subscription1 exists. There is no guarantee provided around messages published prior to the existence of Subscription2. The only exception is that if you were to capture a snapshot on Subscription1 and then seek to that snapshot on Subscription2, then you would see the messages published prior to Subscription2's existence. However, you would only see the messages no older than seven days. Therefore, if the snapshot were more than seven days old, you would see no older messages.

There is no way to extend the retention period beyond 7 days. In general, Kolban's assessment matches the goals of Cloud Pub/Sub: it is a reliable message transport service. The goal of message retention is to ensure that if subscribers are down for a period of time, they will get messages that were sent in that time once they come back up. The same for snapshot and seek: these are particularly useful for the case of restoring from a bad deployment of a subscriber that might have acked messages it should not have. Using seek allows one to replay those messages so they can be processed properly once the subscriber is fixed.

Upvotes: 2

Kolban
Kolban

Reputation: 15276

Messages published to a topic are available to subscriptions taken only prior to those messages being published. If you publish a message at 1pm and take a subscription at 2pm, there is no possibility that any messages published prior to to 2pm will be available for that subscription.

I think of Pub/Sub as a transport and delivery technology as opposed to thinking about it as a storage repository. If you think you may need messages that were previously published before a subscription was taken, consider a design pattern where you add a special subscriber that takes each message published and inserts it into a datastore for subsequent retrieval.

If you now find that you need historical messages, you can query that store for historical items while taking a subscription for new items that will subsequently arrive.

Upvotes: 3

Related Questions