CODEWITHSUNDEEP
apache-kafkaapache-kafka-streamskafka-topic
user13419533
user13419533

Reputation:

Should I create more topics or more partitions?

Kafka gets orders from others countries.

I need to group these orders by countries. Should I create more topics with country name or about to have one topic with different partitions?

Another was it to have one topic and use strean Kafka that filters orders and sends to specific country topic?

What is better if anmount of countries is over 180?

I want distribute orders across executers who is placed in specific country/city.

Remark:

So, order has data about country/city. Then Kafka must find executers in this country/city and send them the same order.

Upvotes: 4

Views: 2266

Answers (1)

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39790

tl;dr

In your case, I would create one topic countries and use the country_id or country_name as the message key so that messages for the same country, are placed in the same partition. In this way, each partition will contain information for specific country (or countries - it depends).

I would say this decision depends on multiple factors;

  • Logic/Separation of Concerns: You can decide whether to use multiple topics over multiple partitions based on the logic you are trying to implement. Normally, you need distinct topics for distinct entities. For example, say you want to stream users and companies. It doesn't make much sense to create a single topic with two partitions where the first partition holds users and the second one holds the companies. Also, having a single topic for multiple partitions won't allow you to implement e.g. message ordering for users that can only be achieved using keyed messages (message with the same key are placed in the same partition).

  • Host storage capabilities: A partition must fit in the storage of the host machine while a topic can be distributed across the whole Kafka Cluster by partitioning it across multiple partitions. Kafka Docs can shed some more light on this:

    The partitions in the log serve several purposes. First, they allow the log to scale beyond a size that will fit on a single server. Each individual partition must fit on the servers that host it, but a topic may have many partitions so it can handle an arbitrary amount of data. Second they act as the unit of parallelism—more on that in a bit.

  • Throughput: If you have high throughput, it makes more sense to create different topics per entity and split them into multiple partitions so that multiple consumers can join the consumer group. Don't forget that the level of parallelism in Kafka is defined by the number of partitions (and obviously active consumers).

  • Retention Policy: Message retention in Kafka works on partition/segment level and you need to make sure that the partitioning you've made in conjunction with the desired retention policy you've picked will support your use case.

Upvotes: 4

Related Questions