Errno 111: Connection refused when connecting to Elasticsearch with Python Script

Question

I got an issue with a dockerized elasticsearch cluster that I would like to connect with a simple python indexing script.

After executing docker-compose up it brings up the cluster, kibana as well, but when it tries to connect to ES via the python script I get the following error:

elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f1c04e0c0d0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f1c04e0c0d0>: Failed to establish a new connection: [Errno 111] Connection refused)

I can reach ES with the browser, I can also index regularly if I execute the script in pycharm. So it has to do with docker, but I can't find the source of the problem. This is my dockerfile:

    version: '3.4'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es01
    environment:
      #- discovery.type=single-node needed?
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.1
    container_name: kib01
    depends_on:
      - es01
      - es02
      - es03
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic

  web:
    build: .
    ports:
      - 8000:8000
    depends_on:
      - es01
      - es02
      - es03
    networks:
      - elastic


volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

And the python script:

from elasticsearch import Elasticsearch, helpers
import sys, json, os
import requests

es = Elasticsearch([{'host': '127.0.0.1', 'port': 9200}])

def load_json(directory):
    " Use a generator, no need to load all in memory"
    for filename in os.listdir(directory):
        if filename.endswith('.json'):
            filename = "JSON/" + filename
            with open(filename,'r') as open_file:
                yield json.load(open_file)

if __name__ == '__main__':
    helpers.bulk(es, load_json("JSON"), index='urteile')

Curl localhost from outside the container works, from inside it does not and gives following error:

A curl command within the docker file also gives me a Connection refused by the way. This is the error I get:

mapping_1  | * TCP_NODELAY set
mapping_1  | * connect to 127.0.0.1 port 9200 failed: Connection refused
mapping_1  | *   Trying ::1...
mapping_1  | * TCP_NODELAY set
mapping_1  | * Immediate connect fail for ::1: Address not available
mapping_1  | *   Trying ::1...
mapping_1  | * TCP_NODELAY set
mapping_1  | * Immediate connect fail for ::1: Address not available
mapping_1  | * Failed to connect to localhost port 9200: Connection refused
mapping_1  | * Closing connection 0
mapping_1  | curl: (7) Failed to connect to localhost port 9200: Connection refused

Anybody knows what could be the problem here?

EDIT:

Okay so the comments with putting es01 instead of localhost helped, in that when inserted into the "client" service it will successfully ping Elasticsearch. Still - the python application tells me that the connection is refused...

Answer 1

You are using localhost for the ES cluster address in your python code. ES is running on separate containers so you need to use those hostnames. In your case, you can use the service name declared in the docker compose yaml file (es01, es02, etc) instead of 127.0.0.1