CODEWITHSUNDEEP
cassandraapache-nifi
apspam
apspam

Reputation: 25

NiFi: Routing flowfiles evenly to different processors

I am trying to use NiFi's PutCassandraRecord to input data into a Cassandra cluster of 3 nodes(provided all 3 nodes address to PutCassandraRecord). The throughput I am getting isn't enough for my workload, so I tested using 3 different PutCassandraRecord processors and providing 1 node address to each processor. I was able to get greater throughput this route.

My question is could I split the amount of incoming flowfiles evenly to 3 or more different processors? More specifically, I am using ListenHTTP to listen for incoming data, and I want to split the incoming data evenly to 3 different PutCassandraRecord processors

I am running NiFi 1.9.2

Upvotes: 1

Views: 645

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18660

You could use the DistributeLoad processor with round-robin distribution.

Also, you could try having a single PutCassandraRecord processor, but increase the concurrent tasks of the processor from 1 to 3. This should be similar to having 3 separate processors, but I don't know anything about Cassandra so I don't know the difference between specifying all nodes vs only one.

Upvotes: 3

Related Questions