CODEWITHSUNDEEP
elasticsearchdocker-composefilebeat
user1574598
user1574598

Reputation: 3881

Index Name Not Being Set in Filebeat to Elasticsearch - ELK .NET Docker ElasticHQ

I am experimenting with some json that has been formatted in accordance with Elasticsearch, so I have gone directly from Filebeat to Elasticsearch, as opposed to going through Logstash. This is using docker-compose:

version: '2.2'
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - discovery.type=single-node
      - cluster.name=docker-
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  filebeat:
    container_name: filebeat
    build:
      context: .
      dockerfile: filebeat.Dockerfile
    volumes:
      - ./logs:/var/log
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
    networks:
      - esnet  
  elastichq:
    container_name: elastichq
    image: elastichq/elasticsearch-hq
    ports:
      - 8080:5000
    environment:
      - HQ_DEFAULT_URL=http://elasticsearch:9200
      - HQ_ENABLE_SSL=False
      - HQ_DEBUG=FALSE
    networks:
      - esnet  
networks:
  esnet:

However, when I open ElasticHQ the index name has been labeled as filebeat-7.5.2-2020.02.10-000001 with a date stamp. I have specified the index name as Sample in my filebeat.yml. Is there something I am missing, or is this behavior normal?

Here is my filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.json
  json.keys_under_root: true
  json.add_error_key: true 

#----------------------------- Elasticsearch output --------------------------------

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample-%{+YYYY.MM.dd}"


setup.template.name: "sample"
setup.template.pattern: "sample-*"

It would be more practical to know something predefined so if I use Postman as opposed to ElasticHQ, I can start querying my data without having to look for the index name.

Upvotes: 0

Views: 387

Answers (1)

chandraP
chandraP

Reputation: 81

I think Filebeat ILM might be taking over instead of the configured index name.

Starting with version 7.0, Filebeat uses index lifecycle management by default when it connects to a cluster that supports lifecycle management. Filebeat loads the default policy automatically and applies it to any indices created by Filebeat.

And when ilm is enabled Filebeat Elasticsearch output index settings are ignored

The index setting is ignored when index lifecycle management is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Configure index lifecycle management to learn how to change the index name.

You might need to disable ILM or better yet configure your desired filename using ILM rollover_alias.

Upvotes: 1

Related Questions