CODEWITHSUNDEEP
google-cloud-pubsub
David Tong
David Tong

Reputation: 101

Job Queue using Google PubSub

I want to have a simple task queue. There will be multiple consumers running on different machines, but I only want each task to be consumed once.

If I have multiple subscribers taking messages from a topic using the same subscription ID is there a chance that the message will be read twice? I've tested something along these lines successfully but I'm concerned that there could be synchronization issues. 

client = SubscriberClient.create(SubscriberSettings.defaultBuilder().build());
subName = SubscriptionName.create(projectId, "Queue");
client.createSubscription(subName, topicName, PushConfig.getDefaultInstance(), 0);

Thread subscriber = new Thread() { 
  public void run() {
    while (!interrupted()) {
      PullResponse response = subscriberClient.pull(subscriptionName, false, 1);
      List<ReceivedMessage> messages = response.getReceivedMessagesList();
      mess = messasges.get(0);
      client.acknowledge(subscriptionName, ImmutableList.of(mess.getAckId()));
      doSomethingWith(mess.getMessage().getData().toStringUtf8());
    }
  }
};
subscriber.start();

Upvotes: 1

Views: 1233

Answers (1)

tobyodavies
tobyodavies

Reputation: 28099

In short, yes there is a chance that some messages will be duplicated: GCP promises at-least-once delivery. Exactly-once-delivery is theoretically impossible in any distributed system. You should design your doSomethingWith code to be idempotent if possible so duplicate messages are not a problem.

You should also only acknowledge a message once you have finished processing it: what would happen if your machine dies after acknowledge but before doSomethingWith returns? your message will be lost! (this fundamental idea is why exactly-once delivery is impossible).

If losing messages is preferable to double processing them, you could add a locking process (write a "processed" token to a consistent database), but this can fail if the write is handled before the message is processed. But at this point you might be able to find a messaging technology that is designed for at-most-once, rather than optimised for reliability.

Upvotes: 4

Related Questions