CODEWITHSUNDEEP
jsonelasticsearchloggingfilebeat
Gary Woodfine
Gary Woodfine

Reputation: 387

Ingesting JSON Logfiles in elasticsearch with filebeats

I have a custom log file in a JSON format, the app we are using will output an 1 entry per file as follows

{"cuid":1,"Machine":"001","cuSize":0,"starttime":"2017-03-19T15:06:48.3402437+00:00","endtime":"2017-03-19T15:07:13.3402437+00:00","rejectcount":47,"fitcount":895,"unfitcount":58,"totalcount":1000,"processedcount":953}

I am trying to ingest this into ElasticSearch. I believe this is possible as I am using ES5.X

I have configured my FileBeat prospector, I have attempted to at least pull out 1 field from the file for now, namely the Cuid

filebeat.prospectors:

input_type: log
json.keys_under_root : true
paths:
C:\Files\output*-Account-*
tags : ["json"]
output.elasticsearch:
# The Logstash hosts
hosts: ["10.1.0.4:9200"]
template.name: "filebeat"
template.path: "filebeat.template.json"
template.overwrite: true

processors:
- decode_json_fields: 
fields: ["cuid"]

When I start the FileBeat , it seems to harvest the files, As I get an entry in the FileBeat Registry files

2017-03-20T13:21:08Z INFO Harvester started for file: 

C:\Files\output\001-Account-20032017105923.json
2017-03-20T13:21:27Z INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=160 publish.events=320 filebeat.harvester.started=160 registrar.states.update=320 registrar.writes=2

However, I can't seem to find the data within Kibana. I am not entirely sure how to find it?

I have ensured the FileBeat templates are loaded in kibana.

I have tried to read the documentation and I think I understand it correctly but I am still very hazy, as I am totally new to the stack.

Upvotes: 0

Views: 1170

Answers (1)

Gary Woodfine
Gary Woodfine

Reputation: 387

I am still not entirely sure if this is the right answer. However I managed to resolve my particular issue. In that we were writing out Multiple JSON files to the directory, all with just single line in, as detailed above. Although FileBeats, appeared to harvest the files, I don't think it was reading them.

I modified the application to make use of log4Net, and implement RollingFileAppender, I then ran the application, which started emiting logs to the directory and if by magic, without modifying the my Filebeat.yml it all just started working.

I can only conclude that Filebeats, does not handle multi one line json files. Unless there is some other configuration I am unaware of.

Upvotes: 0

Related Questions