CODEWITHSUNDEEP
node.jsarchitecturemessage-queuegoogle-cloud-pubsub
Garuuk
Garuuk

Reputation: 2254

Google Pub-Sub two way communication architecture

I'm trying to understand how to do two-way communication with google pub-sub with the following architecture

EDIT: I meant to say subscribers instead of consumers

enter image description here

I'm trying to support the following workflow:

  1. UI sends a request to an api service to process an async process
  2. API Service publishes request to a topic to begin the process kick-off
  3. The consumer picks up the message and processes the async process service.
  4. once the async process service is done it publishes to a process complete topic.
  5. Here is where I want the UI to pick up the process complete message and I'm trying to figure out the best approach.

So two questions:

  1. Is the multiple topic the preferred approach when wanting to do two-way communication back to the client? Or is there a way to do this with a single topic with multiple subscriptions?
  2. How should the consumer of the Process-Complete get the response back to the UI? Should the UI be the consumer of the subscription? Or should I send it back to the api service and publish a websocket message? Both these approaches seem to have tradeoffs.

Upvotes: 0

Views: 757

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17216

Multiple topics are going to be preferred in this situation, one for messages going to the asynchronous processors and then one for the responses that go back. Otherwise, your asynchronous processors are going to needlessly receive the response messages and have to ack them immediately, which is unnecessary extra delivery of messages.

With regard to getting the response back to the UI, the UI should not be the consumer of the subscription. In order to do that, you'd need every running instance of the UI to have its own subscription because otherwise, they would load balance messages across them and you couldn't guarantee that the particular client that sent the request would actually receive the response. The same would be true if you have multiple API servers that need to receive particular responses based on the requests that transmitted through them. Cloud Pub/Sub isn't really designed for topics and subscriptions to be ephemeral in this way; it is best when these are created once and all of the data is transmitted across them.

Additionally, having the UI act as a subscriber means that you'd have to have the credentials in the UI to subscribe, which could be a security issue.

You might also consider not using a topic for the asynchronous response. Instead, you could encode as part of the message the address or socket of the client or API server that expects the response. Then, the asynchronous processor could receive a message, process it, send a response to the address specified in the message, and then ack the message it received. This would ensure responses are routed to where they need to go and minimize the delivery of messages that subscribers just ack that they don't need to process, e.g., messages that were intended for a different API server.

Upvotes: 2

Related Questions