CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-pubsub
George7a
George7a

Reputation: 73

How to know if the pub/sub topic contains messages with specific content?

I have X micro-services each of them are them are publishing to the same pub/sub topic. The messages they publish contain the micro-service ID. I am looking for a way to make sure that the pub/sub has finished servicing all the messages of a certain micro-service. My current solution is to count all the messages that were published from the micro-service, and then count the messages that were served in the pub/sub topic (via cloud-function). There are some known issues with this approach, and I was wondering if there is a better way to know if the pub/sub does not contain messages from the specific micro-service?

Upvotes: 0

Views: 2679

Answers (2)

George7a
George7a

Reputation: 73

The solution I ended up with is as follows:

  1. create a set in redis, and add a new message id to it
  2. Include this new message id in the message published to the pubsub
  3. the cloud function that runs after the pubsub should remove the new message_id from the set
  4. by the end of the process, we check the set and if it is empty then the process is done.

This also handle the times when pub/sub publish the same message more than once.

Upvotes: 0

guillaume blaquiere
guillaume blaquiere

Reputation: 76000

There is no solution to know the content of a message than getting the message, opening it and reading it.

A better solution could be to use a separate topic for each type of message, and is the subscription queue is empty, that means all the message have been processed. If you need to count them, you can use the logs to track your message processing in Cloud FUnctions.

An alternative is to use a single PubSub topic and to add the ID in attribute of the message. Then you can create PubSUb subscription with a filter on that attribute and to oversee the subscription as before.

The bad side of that solution is that it's not compliant with Cloud Functions PubSub integration. You need to create an HTTP functions and a PubSub push subscription to be able to leverage the filter feature.

Upvotes: 2

Related Questions