CODEWITHSUNDEEP
apache-nififlowfile
Agus G
Agus G

Reputation: 1

Apache NiFi Distributed Performance

I have a file which is approximately 40GB in size, my nifi cluster runs on one node, and takes around 60 minutes to decompress. If I have this Processor over 2 or 3 nodes does the processor distribute its work for the decompress time to go down to 30, 25 minutes? Thanks in advance

Upvotes: 0

Views: 191

Answers (1)

Andy
Andy

Reputation: 14194

If the file is a single compressed collection of data, you won't be able to distribute this across multiple nodes because the decompression can't occur in parallel. If this is a file which contains multiple independently-compressed pieces of data that are simply concatenated together, then yes, you could split the data into its component pieces and distribute them to multiple nodes to be decompressed in parallel.

Upvotes: 2

Related Questions