CODEWITHSUNDEEP
elasticsearchdocker-composeelastic-stackelasticsearch-2.0docker-container
Alessandro C
Alessandro C

Reputation: 3560

Configure an ELK cluster in docker containers

I'm trying to configure an ELK cluster using 2 docker containers.

I'm using the following image:

I have created 2 docker containers for that image with docker-compose; each works perfectly in standalone mode.

I want to link the 2 ELK nodes between each other in way to create the cluster, but I haven't found a proper solution. The Elasticsearch node in container1 doesn't communicate with the Elasticsearch node in container2.

These are the two docker-compose.yml:

CONTAINER1:

version: '2'
services:
  elasticsearch01:
    image: sebp/elk:es241_l240_k461
    ports:
      - "5601:5601"
      - "9200:9200"
      - "9300:9300"
      - "5044:5044"
    volumes:
      - /opt/ELK1/logstash/conf.d:/etc/logstash/conf.d
    privileged: true

CONTAINER2:

version: '2'
services:
  elasticsearch02:
    image: sebp/elk:es241_l240_k461
    ports:
      - "5602:5601"
      - "9201:9200"
      - "9301:9300"
      - "5045:5044"
    volumes:
      - /opt/ELK2/logstash/conf.d:/etc/logstash/conf.d
    privileged: true

I'have configured the elasticsearch.yml inside the docker containers in this way:

NODE IN CONTAINER1:

cluster.name: elasticsearchcluster
node.name: node1
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "172.21.0.2"]
discovery.zen.minimum_master_nodes: 1

NODE IN CONTAINER2:

cluster.name: elasticsearchcluster
node.name: node2
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "172.22.0.2"]
discovery.zen.minimum_master_nodes: 1

The key is the discovery.zen.ping.unicast.hosts parameter: I don't have the real IP address, because it's a docker container.

I tried docker inspect elasticsearch01, I have the following "IPAddress" property:

    "NetworkSettings": {
        ...
        "Networks": {
            "ELK1_default": {
                ...
                "Gateway": "172.22.0.1",
                "IPAddress": "172.22.0.2",
                ...
            }
        }
    }

But it doesn't work if I set that IP address.

How to configure the cluster properly?

EDIT

Trying the host ip-address and the port, the node 1 starts, the node 2 fails with no errors.

discovery.zen.ping.unicast.hosts: ["127.0.0.1", "192.168.0.1:9300"] -> OK
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "192.168.0.2:9300"] -> FAILS with no errors

Upvotes: 3

Views: 3953

Answers (2)

Alessandro C
Alessandro C

Reputation: 3560

Thanks to Evaldas Buinauskas, I've found the solution with the stack!

First, we need only one docker-compose.yml. In that file we need to configure two services (one for each container to create), and a network to be shared between the two services.

This is the new docker-compose.yml:

version: '2'
services:
  elk1:
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
    image: sebp/elk:es241_l240_k461
    networks:
      - elk_net
    ports:
      - "5601:5601"
      - "9200:9200"
      - "9300:9300"
      - "5044:5044" 
    volumes:
      - /opt/elk/logstash/conf.d:/etc/logstash/conf.d
    privileged: true
  elk2:
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
    image: sebp/elk:es241_l240_k461
    networks:
      - elk_net
    ports:
      - "5602:5601"
      - "9201:9200"
      - "9301:9300"
      - "5045:5044"
    depends_on:
      - elk1  
    volumes:
      - /opt/elk/logstash/conf.d:/etc/logstash/conf.d
    privileged: true  
networks:
  elk_net:
    driver: bridge

The command docker-compose up will create 3 elements:

  1. the container elk1
  2. the container elk2
  3. the network elk_net

With the docker network inspect elk_net command we can view the (docker) ip addresses assigned to the 2 containers.

The elasticsearch.yml files have to be configured as follows:

cluster.name: elasticsearchcluster
node.name: node1
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: ${IP_ADDRESS_ELK1}
discovery.zen.ping.unicast.hosts: ["${IP_ADDRESS_ELK2}"]
discovery.zen.minimum_master_nodes: 1

cluster.name: elasticsearchcluster
node.name: node2
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: ${IP_ADDRESS_ELK2}
discovery.zen.ping.unicast.hosts: ["${IP_ADDRESS_ELK1}"]
discovery.zen.minimum_master_nodes: 1

With this configuration, the cluster works perfectly: the two nodes are merged correctly and the Http get to each elasticsearch server returns all the documents saved in the 2 nodes.

Upvotes: 1

Evaldas Buinauskas
Evaldas Buinauskas

Reputation: 14077

Instead of using a prepared docker file with ELK stack, you could go for something like this:

version: '3'

services:

  elasticsearch:
    image: elasticsearch:2.4.1
    ports:
      - 9200:9200
    networks:
      - elk

  elasticsearch_slave:
    image: elasticsearch:2.4.1
    networks:
      - elk
    depends_on:
      - elasticsearch
    command: elasticsearch --discovery.zen.ping.unicast.hosts=elasticsearch

  logstash:
    image: logstash:2.3.3
    hostname: logstash
    networks:
      - elk
    volumes:
      - ./logstash.conf:/config/logstash.conf
    depends_on:
      - elasticsearch
    ports:
      - 5044:5044
    command: logstash -f /config/logstash.conf

  kibana:
    image: kibana:4.5.1
    hostname: kibana
    networks:
      - elk
    depends_on:
      - elasticsearch
      - logstash
    ports:
      - 5601:5601

networks:

  elk:
    driver: bridge

Once you start your images using docker-compose up -d, then you can scale slaves using following command docker-compose scale elasticsearch_slave=5

Once that's done - you'll have 5 slaves + client node that opens up port 9200 as a gateway to the whole cluster.

For instance, after doing that, http://localhost:9200/_cat/nodes?v displays following: enter image description here

Upvotes: 2

Related Questions