CODEWITHSUNDEEP
apache-nifi
Holm
Holm

Reputation: 3365

Does the events in the same partition go to the same FlowFile using Kafka Consumer in NiFi

The post sets the Max Poll Records to 1 to guarantee the events in one flow file come from the same partition. https://community.hortonworks.com/articles/223849/simple-backup-and-restore-of-kafka-messages-via-ni.html

Does that mean if using Message Demarcator, the events in the same FlowFile can be from different partitions?

from the source code I think the above thinking is true? https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-9-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L366

Upvotes: 0

Views: 148

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18670

When using a demarcator it creates a bundle per topic/partition, so you will get flow files where all messages are from the same topic partition:

https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-9-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L378

The reason that post set max pool records to 1 was explained in the post, it was because the key of the messages is only available when there is 1 message per flow file, and they needed the key in this case. In general, it is better to not do this and to have many messages per flow file.

Upvotes: 2

Related Questions