When running docker-compose up whith elasticsearch image throwing Java error

Question

Given I have dockerized application that connects to docker image of elasticsearch via docker-compose

e.g.:

---
version: '3'
services:
  elasticsearch:
    image: elasticsearch:6.6.1
    environment:
  myapp:
    image: myapp:latest
    build:
      context: .
      dockerfile: Dockerfile
    links:
      - elasticsearch:elasticsearch

When I do docker-compose up

I get error:

elasticsearch_1         | [2019-03-26T14:46:04,345][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
elasticsearch_1         | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         | Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
elasticsearch_1         |   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   ... 6 more
elasticsearch_1         | Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data/nodes/0
elasticsearch_1         |   at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:215) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   ... 6 more
elasticsearch_1         | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock
elasticsearch_1         |   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
elasticsearch_1         |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
elasticsearch_1         |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
elasticsearch_1         |   at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:178) ~[?:?]
elasticsearch_1         |   at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
elasticsearch_1         |   at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
elasticsearch_1         |   at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
elasticsearch_1         |   at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
elasticsearch_1         |   at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
elasticsearch_1         |   at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:208) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:295) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.6.1.jar:6.6.1]
elasticsearch_1         |   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.6.1.jar:6.6.1]

Answer 1

Don't exactly know why this happens. For me it was due to upgrade of elasticsearch version in docker-compose file. We upgraded from image: elasticsearch:6.5 to image: elasticsearch:6.6.1 and in some developers laptop it left elasticsearch docker images in weird state (mainly those who use OSx /mac for development)

really only thing that helped is to remove the Elasticsearch images and repul them

Try erasing the docker images with

docker-compose down --rmi local

warning: this will remove all images related to the project from your computer https://docs.docker.com/compose/reference/down/

then re-pull images

docker-compose pull
docker-compose up

and it should work

---

if this doesn't help try

$ docker images | grep elast

# elasticsearch                                                6.6.1               c6ffcb0ee97e        5 weeks ago         842MB
# elasticsearch                                                5.6                 34b57bbab755        2 months ago        486MB

docker rmi elasticsearch:6.6.1
docker rmi elasticsearch:5.6