CODEWITHSUNDEEP
apache-flinkapache-nifi
oceansize
oceansize

Reputation: 731

Apache Nifi Site To Site Data Partitioning

I have a single output port in NiFi flow and I have a Flink job that's consuming data from this port using NiFi Site To Site protocol (Flink provides appropriate connector). The consumption is parallel - i.e. there are multiple Flink sources reading from the same NiFi port.

What I would like to achieve is kind of partitioned data load balancing between running Flink sources - i.e. ensure that data with the same key is always delivered to the same Flink source (similar to ActiveMQ message groups or Kafka partitioning). This is needed for ordering purposes.

Unfortunately, I was unable to find any documentation telling how to accomplish that. Any suggestions really appreciated.

Thanks in advance,

Upvotes: 0

Views: 94

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18660

Site-to-site wasn't really made to do what you are asking for. The best way to achieve it would be for NiFi to publish to Kafka, and then Flink consume from Kafka.

Upvotes: 2

Related Questions