CODEWITHSUNDEEP
javapublish-subscribegoogle-cloud-pubsub
Elad Benda
Elad Benda

Reputation: 36654

google pub sub fetches messages in different times

I have a java client of google-pubsub api.

I sent 4 messages at the same time to the pubsub.

My subscriber is triggered once an hour and loop on the pubsub till it's empty.

Nevertheless I see the messages are fetched in different times even though they were all sent at the same time and there was a long delay till the next automated subscriber synchronised fetch.

// A hotfix as cofman is too slow for ack
while (pubSubIsFull) {
    // Fetch messages by order
    log("before fetched messages by order");
    List<ReceivedMessage> messages = mySyncSubscriber.fetch(100000, true);

    messages = messages.stream().sorted(Comparator.comparingLong(message2 -> message2.getMessage().getPublishTime().getSeconds())).collect(Collectors.toList());

    log("fetched " + messages.size() + " messages by order");
    pubSubIsFull = messages.size() > 0;

    log("before processMessages");
    List<String> ackIds = processMessages(messages);
    log("after processMessages");
    mySyncSubscriber.sendAck(ackIds);
}

import com.google.pubsub.v1.PullRequest.Builder;

public List<ReceivedMessage> fetch(int maxMessages, boolean returnImmediately) {
    String subscriptionName = this.getSubscriptionName(this.getSubscriptionId()).toString();
    Builder pullRequestBuilder = PullRequest.newBuilder().setSubscription(subscriptionName).setReturnImmediately(returnImmediately);

    if (maxMessages != 0) {
      pullRequestBuilder.setMaxMessages(maxMessages);
    }

    PullRequest pullRequest = pullRequestBuilder.build();
    PullResponse pullResponse = (PullResponse)this.subscriber.pullCallable().call(pullRequest);

    return pullResponse.getReceivedMessagesList();
}

Is this a bug in pub sub?

Is there something I can do to change it?

Upvotes: 3

Views: 1048

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17206

Cloud Pub/Sub offers no ordering guarantees. If you publish multiple messages--even some time apart--and then later fetch all of the messages, there are no guarantees that the messages published earlier will be received by the subscriber first.

Additionally, receiving 0 messages in a single response is not a good way to determine that no messages are available. Sometimes responses will contain 0 messages even when there are messages available, particularly if returnImmediately is set to true (as noted in my response to your other question). At the very least, you'd want to make sure that several requests over a period of time all return 0 messages before deciding that there are no message available. Even better would be to query the Stackdriver metric for subscription/num_undelivered_messages, which indicates the number of messages not yet acknowledged by your subscription.

Upvotes: 3

Related Questions