Solr 7.4 throws an error when I try to start it

Question

I'm migrating Solr from 3 old machines to 3 new ones and I'm only able to get Solr to run on only 1 of the 3 new machines. Solr was running fine on the 3 old machines, but can't figure out why it won't run on 2 of the new machines.

I'm running Solr 7.4 on 3 RHEL 7 machines.

For the 2 new machines that Solr isn't able to start on, I get the following error when trying to start it.

Starting Solr...

find: `/user/solr/tagCollection/core_node3': No such file or directory

*** [WARN] *** Your open file limit is currently 1024.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 4096.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Waiting up to 180 seconds to see Solr running on port 8983 [-]  Still not seeing Solr listening on 8983 after 180 seconds!

2021-06-22 08:15:48.910 INFO  (qtp867148091-17) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=183
2021-06-22 08:15:48.911 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:16:49.424 INFO  (qtp867148091-17) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=182
2021-06-22 08:16:49.425 INFO  (qtp867148091-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
2021-06-22 08:17:49.932 INFO  (qtp867148091-17) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=186
2021-06-22 08:17:49.934 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:18:50.463 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=188
2021-06-22 08:18:50.466 INFO  (qtp867148091-22) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:19:50.979 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=187
2021-06-22 08:19:50.981 INFO  (qtp867148091-19) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:20:01.494 INFO  (qtp867148091-18) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2021-06-22 08:20:01.495 INFO  (qtp867148091-18) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=1
2021-06-22 08:20:51.495 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=188
2021-06-22 08:20:51.502 INFO  (qtp867148091-14) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=6
2021-06-22 08:21:52.024 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=186
2021-06-22 08:21:52.026 INFO  (qtp867148091-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:22:52.546 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=186
2021-06-22 08:22:52.548 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:23:53.050 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=182
2021-06-22 08:23:53.053 INFO  (qtp867148091-22) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=1
2021-06-22 08:24:53.569 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=188
2021-06-22 08:24:53.570 INFO  (qtp867148091-85) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
2021-06-22 08:25:01.787 INFO  (qtp867148091-18) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2021-06-22 08:25:01.788 INFO  (qtp867148091-18) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=1
2021-06-22 08:25:54.096 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=186
2021-06-22 08:25:54.098 INFO  (qtp867148091-14) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
2021-06-22 08:26:54.617 INFO  (qtp867148091-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=194
2021-06-22 08:26:54.619 INFO  (qtp867148091-19) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=0
2021-06-22 08:27:55.130 INFO  (qtp867148091-19) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.Collection3.shard1.replica_n1:UPDATE./update.requests&key=solr.core.Collection3.shard1.replica_n1:QUERY./select.requests&key=solr.core.Collection3.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=188
2021-06-22 08:27:55.131 INFO  (qtp867148091-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=

Anyone have any ideas on how to resolve this? I've updated all of my config settings to use the new IPs in Zookeeper and when I run a zookeeper status on all 3 new servers, they are reporting accordingly. I'm just really stumped as to only why 1 of the 3 new servers are working.

Also, here is the output that I get on my 1 host where Solr is running fine.

Starting Solr...

find: `/user/solr/tagCollection/core_node3': No such file or directory

*** [WARN] *** Your open file limit is currently 1024.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 1024.
 It should be set to 65000 to avoid operational disruption.
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh
Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require
RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.

Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=10825). Happy searching!
                                                                                                                                                                  
COMMAND   PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    10825 solruser  125u  IPv6 7365290      0t0  TCP *:8983 (LISTEN)

Answer 1

So after spending hours trying to troubleshoot this, we found out that 2 of our new VMs weren't recreated to have the same specs as our old machines. They had about half the RAM than our original machines. So whenever we started Solr on them they never started because as part of the Solr start up process, we set a custom size for the RAM which was a higher value than what the new machines had. After rebuilding the 2 servers to new instances with correct specs, Solr started with no problems.

Answer 2

This is just a warning and you should address it if you will work with big data. So you need to add the limitation of open files in your OS. If you use Linux add the following in /etc/sysctl.conf and restart it after relogging.

soft nofile 65536
hard nofile 65536

More info: Setting the Open File Limit (Linux/Unix)