Filebeat not starting in windows

Question

Facing problem with staring up the Filebeat in windows 10, i have modified the filebeat prospector log path with elasticsearch log folder located in my local machine "E:" drive also i have validated the format of filebeat.yml after made the correction but still am getting below error on start up.

Filebeat version : 6.2.3

Windows version: 64 bit

Filebeat.yml (validated yml format)

filebeat.prospectors:
-
    type: log
    enabled: true
    paths:
        - 'E:\Research\ELK\elasticsearch-6.2.3\logs\*.log'
filebeat.config.modules:
    path: '${path.config}/modules.d/*.yml'
    reload.enabled: false
setup.template.settings:
    index.number_of_shards: 3
setup.kibana:
    host: 'localhost:5601'
output.elasticsearch:
    hosts:
        - 'localhost:9200'
    username: elastic
    password: elastic

Filebeat Startup Log:

E:\Research\ELK\filebeat-6.2.3-windows-x86_64>filebeat --setup -e
2018-03-24T22:58:39.660+0530    INFO    instance/beat.go:468    Home path: [E:\Research\ELK\filebeat-6.2.3-windows-x86_64] Config path: [E:\Research\ELK\filebeat-6.2.3-windows-x86_64] Data path: [E:\Research\ELK\filebeat-6.2.3-windows-x86_64\data] Logs path: [E:\Research\ELK\filebeat-6.2.3-windows-x86_64\logs]
2018-03-24T22:58:39.661+0530    INFO    instance/beat.go:475    Beat UUID: f818bcc0-25bb-4545-bcd4-3523366a4c0e
2018-03-24T22:58:39.662+0530    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.3
2018-03-24T22:58:39.662+0530    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-03-24T22:58:39.665+0530    INFO    pipeline/module.go:76   Beat name: DESKTOP-J932HJH
2018-03-24T22:58:39.666+0530    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-03-24T22:58:39.666+0530    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-03-24T22:58:39.672+0530    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-03-24T22:58:39.672+0530    INFO    kibana/client.go:69     Kibana url: http://localhost:5601
2018-03-24T22:59:08.882+0530    INFO    instance/beat.go:583    Kibana dashboards successfully loaded.
2018-03-24T22:59:08.882+0530    INFO    elasticsearch/client.go:145     Elasticsearch url: http://localhost:9200
2018-03-24T22:59:08.885+0530    INFO    elasticsearch/client.go:690     Connected to Elasticsearch version 6.2.3
2018-03-24T22:59:08.888+0530    INFO    instance/beat.go:301    filebeat start running.
2018-03-24T22:59:08.888+0530    INFO    registrar/registrar.go:108      Loading registrar data from E:\Research\ELK\filebeat-6.2.3-windows-x86_64\data\registry
2018-03-24T22:59:08.888+0530    INFO    registrar/registrar.go:119      States Loaded from registrar: 5
2018-03-24T22:59:08.888+0530    INFO    crawler/crawler.go:48   Loading Prospectors: 1
2018-03-24T22:59:08.889+0530    INFO    log/prospector.go:111   Configured paths: [E:\Research\ELK\elasticsearch-6.2.3\logs\*.log]
2018-03-24T22:59:08.890+0530    INFO    log/harvester.go:216    Harvester started for file: E:\Research\ELK\elasticsearch-6.2.3\logs\elasticsearch.log
2018-03-24T22:59:08.892+0530    ERROR   fileset/factory.go:69   Error creating prospector: No paths were defined for prospector accessing config
2018-03-24T22:59:08.892+0530    INFO    crawler/crawler.go:109  Stopping Crawler
2018-03-24T22:59:08.893+0530    INFO    crawler/crawler.go:119  Stopping 1 prospectors
2018-03-24T22:59:08.897+0530    INFO    log/prospector.go:410   Scan aborted because prospector stopped.
2018-03-24T22:59:08.897+0530    INFO    log/harvester.go:216    Harvester started for file: E:\Research\ELK\elasticsearch-6.2.3\logs\elasticsearch_deprecation.log
2018-03-24T22:59:08.897+0530    INFO    prospector/prospector.go:121    Prospector ticker stopped
2018-03-24T22:59:08.898+0530    INFO    prospector/prospector.go:138    Stopping Prospector: 18361622063543553778
2018-03-24T22:59:08.898+0530    INFO    log/harvester.go:237    Reader was closed: E:\Research\ELK\elasticsearch-6.2.3\logs\elasticsearch.log. Closing.
2018-03-24T22:59:08.898+0530    INFO    crawler/crawler.go:135  Crawler stopped
2018-03-24T22:59:08.899+0530    INFO    registrar/registrar.go:210      Stopping Registrar
2018-03-24T22:59:08.908+0530    INFO    registrar/registrar.go:165      Ending Registrar
2018-03-24T22:59:08.910+0530    INFO    instance/beat.go:308    filebeat stopped.
2018-03-24T22:59:08.948+0530    INFO    [monitoring]    log/log.go:132  Total non-zero metrics  
2018-03-24T22:59:08.948+0530    INFO    [monitoring]    log/log.go:133  Uptime: 29.3387858s
2018-03-24T22:59:08.949+0530    INFO    [monitoring]    log/log.go:110  Stopping metrics logging.
2018-03-24T22:59:08.950+0530    ERROR   instance/beat.go:667    Exiting: No paths were defined for prospector accessing config
Exiting: No paths were defined for prospector accessing config

Answer 1

I understand that this topic is a bit old however looking at the amount of views that this has received at the time of posting this (June 2019), I think it would be safe to add more informations as this is fairly frustrating to get while very easy to fix.

Before I explain what I did, allow me to say I had this problem on a Linux system but the problem/solution should be the same on all plateforms.

After having updated the logback-spring.xml and restarted the service, it kept refusing spitting back the following error:

 ERROR   instance/beat.go:824    Exiting: Can only start an input when all related states are finished: {Id:163850-64780 Finished:false Fileinfo:0xc42016c1a0 Source:/some/path/here/error.log Offset:0 Timestamp:2019-06-13 09:15:35.481163602 -0400 EDT m=+0.107516982 TTL:-1ns Type:log Meta:map[] FileStateOS:163850-64780}

My solution was simply to edit the /etc/filebeat/filebeat.yml and comment as much stuff as I could (Going back to nearly a vanilla/basic configuration).

After having done so, restarting filebeat worked and this ended up being a duplicate path entry with another file somewhere in the system, possibly under the modules.

Answer 2

Check this path ${path.config}/modules.d/ or check by command line "filebeat.exe modules list", if some modules are active, which do not work with windows.

For instance the system.yml (module) does not run on plain windows, because there is no syslog. But the system module is active by default. So you have to disable it first. If I have it enabled, I run in the exactly the same error message, and filebeat stops.

Answer 3

Rewrite the first part of the yml using this format:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

Remove also the empty new line and take attention to the indentation.