CODEWITHSUNDEEP
gcloudgoogle-cloud-pubsub
ZAky
ZAky

Reputation: 1301

process pubsub messages in constant rate. Using streaming and serverless

The scenario: I have thousands of requests I need to issue each day.

I know the number at the beginning of the day and hopefully I want to send all the data about the requests to pubsub. Message per request.

I want to make the requests in constant rate. for example if I have 172800 requests, I want to process 2 in each second.

The ultimate way will involved pubsub push and cloud run.

Using pull with long running instances is also an option.

Any other option are also welcome.

I want to avoid running in a loop and fetch records from a database with limit. This is how I am doing it today.

Upvotes: 1

Views: 851

Answers (1)

Prajna Rai T
Prajna Rai T

Reputation: 1810

You can use batch and flow control settings for fine-tuning Pub/Sub performance which will help in processing messages at a constant rate.

Batching

A batch, within the context of Cloud Pub/Sub, refers to a group of one or more messages published to a topic by a publisher in a single publish request. Batching is done by default in the client library or explicitly by the user. The purpose for this feature is to allow for a higher throughput of messages while also providing a more efficient way for messages to travel through the various layers of the service(s). Adjusting the batch size (i.e. how many messages or bytes are sent in a publish request) can be used to achieve the desired level of throughput.

Features specific to batching on the publisher side include setElementCountThreshold(), setRequestByteThreshold(), and setDelayThreshold() as part of setBatchSettings() on a publisher client (the naming varies slightly in the different client libraries). These features can be used to finely tune the behavior of batching to find a better balance among cost, latency, and throughput.

Note: The maximum number of messages that can be published in a single batch is 1000 messages or 10 MB.

An example of these batching properties can be found in the Publish with batching settings documentation.

Flow Control

Flow control features on the subscriber side can help control the unhealthy behavior of tasks on the pipeline by allowing the subscriber to regulate the rate at which messages are ingested. These features provide the added functionality to adjust how sensitive the service is to sudden spikes or drops of published throughput.

Some features that are helpful for adjusting flow control and other settings on the subscriber are setMaxOutstandingElementCount(), setMaxOutstandingRequestBytes(), and setMaxAckExtensionPeriod().

Examples of these settings being used can be found in the Subscribe with flow control documentation.

For more information refer to this link.

If you are having long running instances as subscribers, then you will need to set relevant FlowControl settings for example .setMaxOutstandingElementCount(1000L)

Once you have set it to the desired number (for example 1000), this should control the maximum amount of messages the subscriber receives before pausing the message stream, as explained in the code below from this documentation:

// The subscriber will pause the message stream and stop receiving more messsages from the
    // server if any one of the conditions is met.
    FlowControlSettings flowControlSettings =
        FlowControlSettings.newBuilder()
            // 1,000 outstanding messages. Must be >0. It controls the maximum number of messages
            // the subscriber receives before pausing the message stream.
            .setMaxOutstandingElementCount(1000L)
            // 100 MiB. Must be >0. It controls the maximum size of messages the subscriber
            // receives before pausing the message stream.
            .setMaxOutstandingRequestBytes(100L * 1024L * 1024L)
            .build();

Upvotes: 2

Related Questions