CODEWITHSUNDEEP
apache-nifi
Manmit
Manmit

Reputation: 45

While starting NIFI.. getting Exception: java.nio.file.FileSystemException: ./state/local/partition-0/107654.journal: Too many open files

While starting NiFi it fails with below error

    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
    at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55)
    ... 37 common frames omitted

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: java.nio.file.FileSystemException: ./state/local/partition-0/107654.journal: Too many open files at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)

i Debuged and found there are alot of file created in all the partition in

/usr/lib/NiFi/NiFi/nifi-1.11.4/state/local

partition-0 partition-11 partition-14 partition-3 partition-6 partition-9 partition-1 partition-12 partition-15 partition-4 partition-7 snapshot partition-10 partition-13 partition-2 partition-5 partition-8 wali.lock

like if any partition folder i select ... suppose i select partition-0 so inside that lots of journal entry i can see like

107018.journal 107632.journal 108246.journal 108860.journal 109474.journal 110088.journal 107019.journal 107633.journal 108247.journal 108861.journal 109475.journal 110089.journal 107020.journal 107634.journal 108248.journal 108862.journal 109476.journal 110090.journal 107021.journal 107635.journal 108249.journal 108863.journal 109477.journal 110091.journal 107022.journal 107636.journal 108250.journal 108864.journal 109478.journal 110092.journal 107023.journal 107637.journal 108251.journal 108865.journal 109479.journal 110093.journal 107024.journal 107638.journal 108252.journal 108866.journal 109480.journal 110094.journal 107025.journal 107639.journal 108253.journal 108867.journal 109481.journal 110095.journal 107026.journal 107640.journal 108254.journal 108868.journal 109482.journal 110096.journal 107027.journal 107641.journal 108255.journal 108869.journal 109483.journal 110097.journal 107028.journal 107642.journal 108256.journal 108870.journal 109484.journal 110098.journal 107029.journal 107643.journal 108257.journal 108871.journal 109485.journal 110099.journal 107030.journal 107644.journal 108258.journal 108872.journal 109486.journal 110100.journal 107031.journal 107645.journal 108259.journal 108873.journal 109487.journal 110101.journal 107032.journal 107646.journal 108260.journal 108874.journal 109488.journal 110102.journal 107033.journal 107647.journal 108261.journal 108875.journal 109489.journal 110103.journal 107034.journal 107648.journal 108262.journal 108876.journal 109490.journal 110104.journal 107035.journal 107649.journal 108263.journal 108877.journal 109491.journal 110105.journal

Entries are even more and i guess this is leading to not start the Nifi .

Hope i find a solution here for this issue

Upvotes: 1

Views: 483

Answers (1)

Andy
Andy

Reputation: 14194

Did you follow the instructions to raise the open file limit on your operating system? NiFi requires a high number of open files when reading/writing to the internal data repositories, so if this is limited, it can often prevent successful operation.

From the docs:

Maximum File Handles

NiFi will at any one time potentially have a very large number of file handles open. Increase the limits by editing /etc/security/limits.conf to add something like

  • hard nofile 50000
  • soft nofile 50000

Maximum Forked Processes

NiFi may be configured to generate a significant number of threads. To increase the allowable number, edit /etc/security/limits.conf

  • hard nproc 10000
  • soft nproc 10000

And your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding

  • soft nproc 10000

Upvotes: 0

Related Questions