solr SolrIndexWriter was not closed prior to finalize(),​ indicates a bug -- POSSIBLE RESOURCE LEAK

Question

I have a Solr cloud (version 7.4) with 2 nodes (each one has 10Gb SSD hard and 256Gb memory and 50Gb heap) and 10 collections. Today I encountered this error about instagram_prime_archive collection :

SolrIndexWriter was not closed prior to finalize(),​ indicates a bug -- POSSIBLE RESOURCE LEAK!!!

Error closing IndexWriter

null:org.apache.lucene.index.IndexNotFoundException: no segments* file found in LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/home/solr/data/instagram_prime_archive_shard1_replica_n15/data/index.20190313041339667 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b54ef37; maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [_14jqg.cfe, _14jqg.cfs, _14jqg.si, _14jqg_xcc.liv, _17wrq.cfe, _17wrq.cfs, _17wrq.si, _17wrq_8im.liv, _19qhp.cfe, _19qhp.cfs, _19qhp.si, _19qhp_3ik.liv, _19wns.cfe, _19wns.cfs, _19wns.si, _19wns_1yp.liv, _1a0lx.cfe, _1a0lx.cfs, _1a0lx.si, _1a0lx_9a.liv, _1a3u8.cfe, _1a3u8.cfs, _1a3u8.si, _1a3u8_7g.liv, _1a5fe.cfe, _1a5fe.cfs, _1a5fe.si, _1a5fe_es.liv, _1a7fu.cfe, _1a7fu.cfs, _1a7fu.si, _1a7fu_bg.liv, _1a9oo.cfe, _1a9oo.cfs, _1a9oo.si, _1a9oo_cv.liv, _1abp0.cfe, _1abp0.cfs, _1abp0.si, _1abp0_3fk.liv, _1afn9.cfe, _1afn9.cfs, _1afn9.si, _1afn9_98.liv, _1ajhu.cfe, _1ajhu.cfs, _1ajhu.si, _1ajhu_6r.liv, _1an7f.cfe, _1an7f.cfs, _1an7f.si, _1an7f_6t.liv, _1ap2m.cfe, _1ap2m.cfs, _1ap2m.si, _1ap2m_87.liv, _1au5v.cfe, _1au5v.cfs, _1au5v.si, _1au5v_72.liv, _1audy.cfe, _1audy.cfs, _1audy.si, _1audy_m.liv, _1aup4.cfe, _1aup4.cfs, _1aup4.si, _1aup4_m.liv, _1auza.cfe, _1auza.cfs, _1auza.si, _1auza_x.liv, _1avb8.cfe, _1avb8.cfs, _1avb8.si, _1avb8_u.liv, _1avgt.cfe, _1avgt.cfs, _1avgt.si, _1avgt_k.liv, _1avhd.cfe, _1avhd.cfs, _1avhd.si, _1avhd_8c.liv, _1avq1.cfe, _1avq1.cfs, _1avq1.si, _1avq1_13.liv, _1azqq.cfe, _1azqq.cfs, _1azqq.si, _1azqq_4.liv, _1azwu.cfe, _1azwu.cfs, _1azwu.si, _1azwu_1r.liv, _1b08s.cfe, _1b08s.cfs, _1b08s.si, _1b08s_5.liv, _1b0kq.cfe, _1b0kq.cfs, _1b0kq.si, _1b0kq_m.liv, _1b0ni.cfe, _1b0ni.cfs, _1b0ni.si, _1b0qa.cfe, _1b0qa.cfs, _1b0qa.si, _1b0tm.cfe, _1b0tm.cfs, _1b0tm.si, _1b0tm_1.liv, _1b0zh.cfe, _1b0zh.cfs, _1b0zh.si, _1b100.cfe, _1b100.cfs, _1b100.si, _1b10k.cfe, _1b10k.cfs, _1b10k.si, _1b11e.cfe, _1b11e.cfs, _1b11e.si, _1b11p.cfe, _1b11p.cfs, _1b11p.si, _1b11y.cfe, _1b11y.cfs, _1b11y.si, _1b128.cfe, _1b128.cfs, _1b128.si, _1b12s.cfe, _1b12s.cfs, _1b12s.si, _1b132.cfe, _1b132.cfs, _1b132.si, _1b13c.cfe, _1b13c.cfs, _1b13c.si, _1b13d.dii, _1b13d.dim, _1b13d.fdt, _1b13d.fdx, _1b13d.fnm, _1b13d.nvd, _1b13d.nvm, _1b13d.si, _1b13d_Lucene50_0.doc, _1b13d_Lucene50_0.pos, _1b13d_Lucene50_0.tim, _1b13d_Lucene50_0.tip, _1b13d_Lucene70_0.dvd, _1b13d_Lucene70_0.dvm, _1b13e.dii, _1b13e.dim, _1b13e.fdt, _1b13e.fdx, _1b13e.fnm, _1b13e.nvd, _1b13e.nvm, _1b13e.si, _1b13e_Lucene50_0.doc, _1b13e_Lucene50_0.pos, _1b13e_Lucene50_0.tim, _1b13e_Lucene50_0.tip, _1b13e_Lucene70_0.dvd, _1b13e_Lucene70_0.dvm, _1b13f.dii, _1b13f.dim, _1b13f.fdt, _1b13f.fdx, _1b13f.fnm, _1b13f.nvd, _1b13f.nvm, _1b13f.si, _1b13f_Lucene50_0.doc, _1b13f_Lucene50_0.pos, _1b13f_Lucene50_0.tim, _1b13f_Lucene50_0.tip, _1b13f_Lucene70_0.dvd, _1b13f_Lucene70_0.dvm, _1b14j.dii, _1b14j.dim, _1b14j.fdt, _1b14j.fdx, _1b14j.fnm, _1b14j.nvd, _1b14j.nvm, _1b14j.si, _1b14j_2.liv, _1b14j_Lucene50_0.doc, _1b14j_Lucene50_0.pos, _1b14j_Lucene50_0.tim, _1b14j_Lucene50_0.tip, _1b14j_Lucene70_0.dvd, _1b14j_Lucene70_0.dvm, _1b14k.dii, _1b14k.dim, _1b14k.fdt, _1b14k.fdx, _1b14k.fnm, _1b14k.nvd, _1b14k.nvm, _1b14k.si, _1b14k_2.liv, _1b14k_Lucene50_0.doc, _1b14k_Lucene50_0.pos, _1b14k_Lucene50_0.tim, _1b14k_Lucene50_0.tip, _1b14k_Lucene70_0.dvd, _1b14k_Lucene70_0.dvm, _1b14q.dii, _1b14q.dim, _1b14q.fdt, _1b14q.fdx, _1b14q.fnm, _1b14q.nvd, _1b14q.nvm, _1b14q.si, _1b14q_2.liv, _1b14q_Lucene50_0.doc, _1b14q_Lucene50_0.pos, _1b14q_Lucene50_0.tim, _1b14q_Lucene50_0.tip, _1b14q_Lucene70_0.dvd, _1b14q_Lucene70_0.dvm, _pqyd.cfe, _pqyd.cfs, _pqyd.si, _pqyd_3bhn.liv, _r9bx.cfe, _r9bx.cfs, _r9bx.si, _r9bx_33sj.liv, write.lock]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:832)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:124)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:97)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:131)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:344)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:284)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1168)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.SignatureUpdateProcessorFactory$SignatureUpdateProcessor.processAdd(SignatureUpdateProcessorFactory.java:193)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
at java.lang.Thread.run(Thread.java:748)

I checked the htop and found that the number of threads was 9897 !!!

There was also a lot of threads on the Thread Dump screen :

What can be the reason for these Errors? And why has all this thread been created?

Could my code have a problem?

Answer 1

It may be that SOLR is running using Java8 and the Oracle DB driver you are using does not support it (i.e. 'ojdbc10.jar'). Check the Oracle DB driver installed in the "server/lib" path; if SOLR is running with Java8 then you should have 'ojdbc8.jar' installed.

Answer 2

Follow the steps to resolve issue:

Step 1: cd into your solr directory

Step 2: then go to your core data index directory

cd server/solr/<your-core-name>/data/index/

Step 3: then remove the lock file that is stopping the process

sudo rm write.lock

Step 4: cd into your solr directory again

Step 5: Now restart the solr instance by running following command

bin/solr restart

All Done! Now you can index again without problem