CODEWITHSUNDEEP
elasticsearchfilebeatlog-shipping
user15937765
user15937765

Reputation: 309

Filebeat | how to define a unique index for different log paths?

I am collecting logs from two different paths:

I wanted a that files from each of the paths will be sent to different indexes in elasticsearch. I tried to define the filebeat configuration as follow:

    output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "agent-logs"
      when:
       contains:
        log.file.path: "/var/log/agents/*.log"
    - index: "container-logs"
      when:
       contains:
        log.file.path: "/var/log/containers/*.log"

I also tried:

    output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "agent-logs"
      when.contains:
        log.file.path: "/var/log/agents/*.log"
    - index: "container-logs"
      when.contains:
        log.file.path: "/var/log/containers/*.log"

and:

    output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "agent-logs"
      when.equals:
        log.file.path: "/var/log/agents/*.log"
    - index: "container-logs"
      when.equals:
        log.file.path: "/var/log/containers/*.log"

But nothing seems to be working. Please help!

thank you

Upvotes: 0

Views: 364

Answers (2)

user15937765
user15937765

Reputation: 309

Thank you so much for your answer! But I have another issue... logs from /var/log/agents/*.log are not even passing to elasticsearch although they exist in the path, you know what might be the reason?

this is the full config file:

   filebeat.autodiscover:
     providers:
      - type: kubernetes
        hints.enabled: true
        hints.default_config:
          enabled: false
          type: container
          paths:
            - /var/log/containers/*.log  # CRI path
            - /var/log/agents/*.log

  output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "agent-logs"
      when:
       contains:
        log.file.path: "agents"
    - index: "container-logs"
      when:
       contains:
        log.file.path: "containers"

and I also tried this config, but when I run this config I only get agent logs:

  filebeat.autodiscover:
    providers:
      - type: kubernetes
        hints.enabled: true
        hints.default_config:
          enabled: false
          type: container
          paths:
            - /var/log/containers/*.log
  filebeat.inputs:
   - type: filestream
     id: agent-filestream
     paths:
      - "/var/log/agents/*.log"

  output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "container-logs"
      when:
       contains:
        log.file.path: "containers"
    - index: "agent-logs"
      when:
       contains:
        log.file.path: "agents"

what do you think?

Upvotes: 0

Paulo
Paulo

Reputation: 10346

TLDR;

equals and contains do not support glob patterns. You may want to look into regexp instead ? or another distinct value.

Solution

This might work better:

output.elasticsearch:
    protocol: http
    hosts: ["elasticsearch:9200"]
    compression_level: 1
    indices:
    - index: "agent-logs"
      when:
       contains:
        log.file.path: "agents"
    - index: "container-logs"
      when:
       contains:
        log.file.path: "containers"

Upvotes: 1

Related Questions