CODEWITHSUNDEEP
apache-kafkaapache-flinkflink-streaming
sparkless
sparkless

Reputation: 305

How to send data as key/value to Kafka using Apache Flink with Specific Partitioner

I have a payload in Flink like below;

{
    "memberId": 4
    "total": 5
}

I want to send data to kafka as key-value format with specified partitioner. For partitioner, i'll use Modulo partitioner.

An example for modulo partitioner;

partitionId = value % numPartitions

Let's assume numPartitions parameter is 3. If we can use memberId of the payload defined above, partitionId should be 4 % 3 = 1

According to the above partitioner, I want to send data that have the same partitionId to the same kafka topic. Another examples;

If(assume numPartitions = 3);

memberId: 3 => (3 % 3) => partitionId = 0 => kafka partition 1
memberId: 8 => (8 % 3) => partitionId = 2 => kafka partition 2
memberId: 2 => (2 % 3) => partitionId = 2 => kafka partition 2
memberId: 6 => (6 % 3) => partitionId = 0 => kafka partition 1
memberId: 7 => (7 % 3) => partitionId = 1 => kafka partition 2

If I'm not wrong, flink kafka producer uses FlinkFixedPartitioner if we can not specify any key and partition function. If we set partition function as null, flink kafka producer will use round robin distribution. But i don't know how can i send data to kafka as key/value format, how can i partition it by modulo. How can i achieve that?

Upvotes: 0

Views: 1110

Answers (1)

David Anderson
David Anderson

Reputation: 43439

If you use a KafkaSerializationSchema, then you can create Kafka ProducerRecords, and set the Kafka key (and value). You can also set the partition in the ProducerRecord.

Upvotes: 2

Related Questions