CODEWITHSUNDEEP
apache-pulsar
crak
crak

Reputation: 1665

Is creating a lot of topics with pulsar is a good scenario?

Is that possible and good scenario with pulsar to create a topic ( or a partition ) for each hash on the fly, and delete the topics (or partition ) when it is no more used?

The idea is to be able to read data with the same hash in an ordered fashion without having messages with another hash between two message with the same hash. To allow the customer to keep just a limited amount of aggregating messages in memory.

The consumer should also be able to consume totally one topic (or partition) before starting to consume another one.

As result, the goal is to be able to consume and produce data in different order.

produce  in this order          and read like this
1 2 3 4 5 
_ _ _ _ _
a b c d e                       1 [a b c d e] 
a b c d e                       2 [a b c d e]
a b c d e     -------->         3 [a b c d e]
z y x w v                       4 [z y x w v]
g h i j k                       5 [g h i j k]
_ _ _ _ _

in this exemple the message key hash are of course not shown (heach line have the same key hash )

Upvotes: 5

Views: 1823

Answers (1)

Matteo Merli
Matteo Merli

Reputation: 795

Is that possible and good scenario with pulsar to create a topic ( or a partition ) for each hash on the fly, and delete the topics (or partition ) when it is no more used?

Sure, creating many topics in Pulsar is a much cheaper operation compared to other messaging systems. Topics can be explicitly deleted, or they are automatically deleted when all producers and consumers are disconnected and all subscriptions on the topic were deleted as well.

The consumer should also be able to consume totally one topic (or partition) before starting to consume another one.

If you use 1 topic per key, then on the consumer side you have all the flexibility to decide how to consume the messages.

The only thing you need to make sure is to create subscriptions on all the topics, before publishing messages on them, in order to have Pulsar retaining the data.

Upvotes: 5

Related Questions