CODEWITHSUNDEEP
spring-bootgoogle-cloud-platformgoogle-cloud-pubsub
Jonathan Pereira
Jonathan Pereira

Reputation: 31

GCP PubSub Spring Boot repeat extract message

I need help with a problem with gcp pub/sus. I have a process that send 100 messages with filters to pubsub and another application (in spring boot) receive these messages. When spring boot application receive message from pubsub (not pull), process 100 messages but, into the process, receive more messages, in diferents times receive diferents numbers of messages, any times receive 120, another 140, and the others more than 200. I wasn't found any solution of this, this is my code:

    @Bean
    @ServiceActivator(inputChannel = "pubsubInputChannel")
    public MessageHandler messageReceiver() {
        return message -> {
            System.out.println("Message arrived! Payload: " + new String((byte[]) message.getPayload()));
            //other process of app (call other api)
            AckReplyConsumer consumer = (AckReplyConsumer) message.getHeaders().get(GcpPubSubHeaders.ACKNOWLEDGEMENT);
            consumer.ack();
        };
    }

please help me!!!

Upvotes: 3

Views: 1740

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17206

Duplicate messages can happen for different reasons in Google Cloud Pub/Sub. One thing to keep in mind is that Cloud Pub/Sub offers at-least-once delivery meaning that some amount of duplicates is always possible, so your application must be resilient to them. That many duplicates does seem a bit high, though. In general duplicates can generally happen for the following reasons:

  1. Messages are being sent by the publisher more than once. This can happen if the publisher got disconnected from Cloud Pub/Sub and sent the same message again. If this type of duplication occurs, then the messages will have different message IDs.
  2. The subscriber is taking too long to acknowledge messages. In your code, you have //other process of app (call other api). How long does this process take? If it is longer than the deadline for acknowledging the message, then the message will be redelivered. Keep in mind that if this other process requires locks be grabbed for all messages, there could be a contention issue with too many requests trying to get those locks at the same time, resulting in processing delays. By default, the ack deadline for a message is ten seconds. When using the Java client library, the deadline is automatically extended by the maxAckExtensionPeriod, which defaults to one hour. This property can be set in the DefaultSubscriberFactory for Spring as well.
  3. Messages are not acked at all. If an exception prevents the call to ack or there is deadlock resulting in that line of code never being reached, then the message will be redelivered.
  4. The use case is one of a large backlog of small messages. In this situation, buffers are prone to fill up in the client in a way that results in redelivery of messages.

Upvotes: 1

Related Questions