CODEWITHSUNDEEP
google-cloud-platformpublish-subscribegoogle-cloud-pubsub
shadow
shadow

Reputation: 41

How to publish multiple messages in google Pub/Sub fast?

How to publish multiple messages to pubsub fast? Without multiprocessing and multithreading becaue the code is already in a Thread

The code bellow is publishing 40 messages per second

publisher = pubsub.PublisherClient(
    credentials=credentials,
    batch_settings=types.BatchSettings(
         max_messages=1000,  # default is 100
        max_bytes=1 * 1000 * 1000,  # 1 MiB
        max_latency=0.1,  # default is 10 ms
    )
)

topic_name = 'projects/{project_id}/topics/{topic}'.format(
    project_id=PROJECT_ID,
    topic=TOPIC_PUBSUB,
)

for data in results:
    bytes_json_data = str.encode(json.dumps(data))
    future = publisher.publish(topic_name, bytes_json_data)
    future.result()

Upvotes: 2

Views: 4781

Answers (2)

shadow
shadow

Reputation: 41

Take out:

future.result()

Leave it just:

for data in results:
    bytes_json_data = str.encode(json.dumps(data))
    future = publisher.publish(topic_name, bytes_json_data)

And should take it less than a second to publish 10k messages

Upvotes: 1

Samarth Singal
Samarth Singal

Reputation: 356

Instead of publishing the messages one at a time and then wait on the future, you should publish them all at once and then wait on the published futures at the end. It will look something like:

from concurrent import futures
...
publish_futures = []
for data in results:
    bytes_json_data = str.encode(json.dumps(data))
    future = publisher.publish(topic_name, bytes_json_data)
    publish_futures.append(future)
...
futures.wait(publish_futures, return_when=futures.ALL_COMPLETED)

There's a detailed example in the docs with sample code.

Upvotes: 6

Related Questions