CODEWITHSUNDEEP
elasticsearchdocker-composekibana
zipline86
zipline86

Reputation: 661

How to I have Elastic Search (with multiple nodes) and Kibana in one docker compose file?

I have successfully installed a 3-node Elastic Search cluster with a docker compose file from the Elastic Search web site with no problems elastic search link. I am trying to add Kibana to the .yml file so I can run it all with docker-compose up and was looking at this Elastic Search Kibana install site to try to figure it out what to add Kibana install site. When I try to start the file I made, I get this error kibana_1 | {"type":"log","@timestamp":"2021-02-13T11:20:31Z","tags":["error","elasticsearch","data"],"pid":8,"message":"[ConnectionError]: getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}. When I open the http://localhost:5601, it says Kibana server not ready yet. Can somebody please help me get this working? I marked the section I added in the .yml file.

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.11.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.11.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.11.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

##############################
# My attempt at adding Kibana to the docker file. This file works 
# fine if commenting out this whole section.
  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.2
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - 5601:5601
##############################

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

Upvotes: 4

Views: 8471

Answers (2)

nikhil
nikhil

Reputation: 301

Make sure Docker Engine is allotted at least 4GiB of memory.

docker-compose.yml

version: '3.7'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=password
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=password  
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=password
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - elastic

  kibana:
    image: kibana:7.9.2
    ports:
      - '5601:5601'
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
 



  logstash:
    image: logstash:7.9.2
    ports:
      - '5000:5000'
    volumes:
      - type: bind
        source: ./logstash_pipeline/
        target: /usr/share/logstash/pipeline
        read_only: true


networks:
  elastic:
    driver: bridge

kiban.yml

server.name: kibana
server.host: 0.0.0.0
server.port: 5601
elasticsearch.hosts: [ "http://<ELK server ip>:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

## X-Pack security credentials
#
elasticsearch.username: elastic
elasticsearch.password: password

Create a directory name logstash_pipeline and within that directory create a file beats.conf

beats.conf

input {
    beats {
        port => 5044
    }

}

## Add your filters / logstash plugins configuration here

output {
    elasticsearch {
        hosts => "127.0.0.1:9200"
                index => "filebeat-%{+yyyy.MM.dd}"
        user => "elastic"
        password => "password"
        ecs_compatibility => disabled
    }
}

following requirements and recommendations apply when running Elasticsearch in Docker in production.

Set vm.max_map_count to at least 262144edit The vm.max_map_count kernel setting must be set to at least 262144 for production use.

How you set vm.max_map_count depends on your platform:

The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:

grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144

To apply the setting on a live system, run:

sysctl -w vm.max_map_count=262144

Run docker compose to bring up the three-node Elasticsearch cluster and Kibana

docker-compose up

Upvotes: 5

Triet Doan
Triet Doan

Reputation: 12083

As far as I see, there are 2 problems in your docker-compose file.

  1. Kibana is not in the elastic network.
  2. In the Kibana configuration, you set ELASTICSEARCH_HOSTS=http://elasticsearch:9200. However, none of your Elasticsearch containers is named elasticsearch.

The correct configuration should be somehow like this:

kibana:
    image: docker.elastic.co/kibana/kibana:7.10.2
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    networks:
      - elastic

Upvotes: 1

Related Questions