CODEWITHSUNDEEP
filebeat
Emanuel
Emanuel

Reputation: 13

Filebeat not read all logs from directory

I am configuring filebeat to send to elastic logs located in /var/log/myapp/batch_*

Here my filebeat configuration:

# Version

filebeat version 7.11.0 (amd64), libbeat 7.11.0 [84c4d4c4034fcb49c1a318ccdc7311d70adee15b built 2021-02-08 22:42:11 +0000 UTC]

# Filebeat config

logging.metrics.period: 1h
logging.to_files: true
logging.files:
  rotateeverybytes: 16777216
  keepfiles: 7
  permissions: 0600

filebeat.inputs:
- type: log
  enabled: true
  scan_frequency: 5m
  paths:
    - "/var/log/myapp/batch_*"

output.elasticsearch:
  hosts: ["server:9200"]
  index: "log_test_app-%{+yyyy.MM.dd}"

setup.ilm.enabled: false
setup.template.name: "log_test_app"
setup.template.pattern: "log_test_app-*"
setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 1

I only see that two logs are being sent and within the established directory there are a total of eight logs:

2022-05-24T19:39:55.904Z        INFO    log/input.go:157        Configured paths: [/var/log/myapp/batch_*]
2022-05-24T19:39:55.904Z        INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 3328309751929357009)
2022-05-24T19:39:55.904Z        INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 1
2022-05-24T19:44:55.905Z        INFO    log/harvester.go:302    Harvester started for file: /var/log/myapp/batch_emails.log
2022-05-24T19:44:55.905Z        INFO    log/harvester.go:302    Harvester started for file: /var/log/loyalty/batch_import.log

I show you a list of directory files:

ls -l /var/log/loyalty/batch_*
-rw-r--r-- 1 batch batch   154112 May 24 03:20 /var/log/myapp/batch_gifts.log
-rw-r--r-- 1 batch batch   112319 May 24 02:30 /var/log/myapp/batch_http.log
-rw-r--r-- 1 batch batch  7575342 May 24 02:30 /var/log/myapp/batch_vouchers.log
-rw-r--r-- 1 batch batch  4847849 May 24 19:30 /var/log/myapp/batch_ftp.log
-rw-r--r-- 1 batch batch    99413 May 24 03:40 /var/log/myapp/batch_category.log
-rw-r--r-- 1 root  root    367207 May 24 19:50 /var/log/myapp/batch_emails.log
-rw-r--r-- 1 batch batch      479 Jan  1 23:00 /var/log/myapp/batch_history.log
-rw-r--r-- 1 batch batch  2420916 Jan  1 23:00 /var/log/myapp/batch_lists.php
-rw-r--r-- 1 batch batch 25779499 May 24 19:50 /var/log/myapp/batch_import.log

Is there something wrong with my setup? I tried using the ignore_older parameter: 36h but only two log files are processed.

Thanks for the help.

Upvotes: 0

Views: 1164

Answers (1)

sudhagar ramesh
sudhagar ramesh

Reputation: 124

Welcome to stack overflow, Emanuel :)

I believe you are reading only the log files (*.log) thus you can mention the same.

filebeat.inputs:
- type: log
  enabled: true
  scan_frequency: 5m
  paths:
    - "/var/log/myapp/batch_*.log"

Keep Posted!!! Thanks!!!

Upvotes: 0

Related Questions