CODEWITHSUNDEEP
apache-kafkaquarkussmallrye
user5260143
user5260143

Reputation: 1098

how to define the partitioned by type with smallrye-kafka?

I have a simple quarkus project that create topic "databases" and send list of databases to it.

How can I control the partitions?

I want to have different partition per each DatabaseID so all the messages with same Id will go to the same partition.

here is my application.properties:

kafka.bootstrap.servers=localhost:9092

mp.messaging.outgoing.databases.connector=smallrye-kafka
mp.messaging.outgoing.databases.key.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.databases.value.serializer=com.secupi.dbmanager.model.DatabaseSerializer
quarkus.class-loading.removed-resources."org.slf4j\:slf4j-api"=org.slf4j.Logger.class

and my code

@Outgoing("databases")
public <T> Multi<Record<String, Database>> generate() {
                Collection<Database> dbs = databaseDao.getAll(new FilterParams());
            return Multi.createFrom().items(dbs.stream().map(db -> Record.of(db.getName(), db)));                   
}

By default as I see the topic has been created with 100 partitions.

Thanks

Upvotes: 0

Views: 443

Answers (1)

Ozan G&#252;nalp
Ozan G&#252;nalp

Reputation: 474

The number of partitions in your topic depends on how you are creating it. If it is auto-created the default number of partitions is configured on Kafka broker config. IF you are in dev-mode using Quarkus devservices you can configure it to create topic-partitions for you : https://quarkus.io/guides/kafka#configuring-kafka-topics

If the topic is already created with 100 partitions, all you need to do is produce Kafka records with consistent keys. In the snippet you've sent if you create Record.of with database ID as your key the default partitioner will assign it to a unique partition every time.

Upvotes: 0

Related Questions