Solr 6.x cannot create collection on one Debian Jessie 8.6

Question

Trying to install Solr 6.2.1 on a Debian Jessie I hit a roadblock whereby I am not able to create a collection. The sequence of commands

unzip solr-6.2.1.zip
export JAVA_HOME=/opt/java64/jdk1.8.0_101
./solr-6.2.1/bin/solr -c
./solr-6.2.1/bin/solr create_collection -c hktesting

could not be more innocent and works on another Debian Jessie as well as on an Ubuntu 16.10. On this machine, however, the create_collection runs into a timeout. I even used a freshly added user without any dotfile customizations.

The full log is quite long, so I try to pick the lines I think are relevant. It all starts nicely with:

2016-11-03 08:08:36.360 INFO  (qtp110456297-20) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&maxShardsPerNode=1&collection.configName=hktesting&name=hktesting&action=CREATE&numShards=1&wt=json and sendToOCPQueue=true

Messages follow which all look ok showing a lot going-ons in creating the collection. It all comes to an intermediate end with:

2016-11-03 08:08:38.399 WARN  (qtp110456297-18) [c:hktesting s:shard1 r:core_node1 x:hktesting_shard1_replica1] o.a.s.c.SolrCore [hktesting_shard1_replica1] Solr index directory '/home/badsolr/tmp/solr-6.2.1/server/solr/hktesting_shard1_replica1/data/index' doesn't exist. Creating new index...
2016-11-03 08:08:38.408 INFO  (qtp110456297-18) [c:hktesting s:shard1 r:core_node1 x:hktesting_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /home/badsolr/tmp/solr-6.2.1/server/solr/hktesting_shard1_replica1/data/index

which still looks ok to me. Then comes a three minute hole in the log with nothing happening after which a failure is reported:

2016-11-03 08:11:36.373 ERROR (qtp110456297-20) [   ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:289)
at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:440)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)

This machine has kerberos authentication enabled, which is about the only possibly relevant difference to the other machines I would know of (but don't jump to conclusions).

Answer 1

It turned out that a mount on /media was unresponsive. Even an ls /media hung forever. With strace I could see that Solr got stuck when it tried to access /media. In the strace log I could see that Solr first read in mtab and then steps into /media. Not that I would know why Solr should care about mtab and this mount point, but after getting the mount fixed, Solr started to work normal.