CODEWITHSUNDEEP
apache-nifi
Vinayak Mishra
Vinayak Mishra

Reputation: 371

Nifi content repository not cleaning up and keeps accumalating over time

Nifi content repository works in sync with the Nifi Status bar for sometime after a restart, Nifi version 1.9.1. After that, it starts growing gradually and comes down again only on a restart.

I have a mix of small and large files and are merging files to create big files in the flow. I am potentially hitting a bug https://issues.apache.org/jira/browse/NIFI-6150 which is not working for probably the below mentioned three scenarios:

1) Having both small and big files.

2) Merging small files to create big files. I have a few big files which only go after a restart.

3) Merge content without any failures holds onto flow files for days after the flow file has left flow.

Data flow for this Nifi is 24 TB per day and restart is causing huge lags. Also, I have decompiled and have seen the code is present for the fix.

Upvotes: 3

Views: 4598

Answers (1)

Joe Witt
Joe Witt

Reputation: 2172

It isn't clear what version of NiFi you're running but first I'd recommend getting on the latest release immediately (1.11.4). If you're running at 290MB/sec as implied there hopefully this is on a small cluster of nodes or one well configured and properly resourced node.

But as noted in the JIRA you need a version that has that fix and possibly another for the optimal behavior.

Upvotes: 3

Related Questions