Logstash starts but it does not load the data to elasticsearch

Question

I am trying to read the information in json format from log file and load it in elasticsearch. I am using logstash.

OS: windows 10 ElastiSearch version is 7.6.2 LogStash version is 7.6.2

The log file content is like below:

{"@timestamp":"2020-05-03T15:09:38.255+02:00","@version":1,"message":"The following profiles are active: default","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}
{"@timestamp":"2020-05-03T15:09:59.136+02:00","@version":1,"message":"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}

Below it is the logstash.conf file

input {
    file {
        path => "C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log"
        start_position => "beginning"
        sincedb_path => "NUL"
    }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logback-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

I run the logstash like below :

logstash.bat -f C:\Users\User\Desktop\Apps\logstash-7.6.2\config\logstash.conf

And the output is like below:

[2020-05-03T17:52:08,394][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-05-03T17:52:08,456][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-05-03T17:52:08,519][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-05-03T17:52:08,534][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/Users/User/Desktop/Apps/logstash-7.6.2/config/logstash.conf"], :thread=>"#<Thread:0x6e5e96fc run>"}
[2020-05-03T17:52:08,597][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-05-03T17:52:10,327][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-03T17:52:10,380][INFO ][filewatch.observingtail  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-03T17:52:10,411][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-05-03T17:52:10,929][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

It seeems that the logstash starts correctly but it fails to insert the data in elasticsearch.

I really would appriciate any help.

Regards

P.S

I added output { stdout { codec => json } } and below it is result:

{"message":"{\"@timestamp\":\"2020-05-03T15:24:33.059+02:00\",\"@version\":1,\"message\":\"findAll called\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-1\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"042059d1825b87b6\",\"spanId\":\"042059d1825b87b6\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"042059d1825b87b6\",\"X-B3-TraceId\":\"042059d1825b87b6\"}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"@timestamp\":\"2020-05-03T15:09:34.106+02:00\",\"@version\":1,\"message\":\"Employee Application Started\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.389Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"@timestamp\":\"2020-05-03T15:25:04.625+02:00\",\"@version\":1,\"message\":\"countByDepartmentId called, departmentId: 1\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-2\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"8b54f9e340a458a2\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"8b54f9e340a458a2\",\"X-B3-ParentSpanId\":\"30113ef7e385207d\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"30113ef7e385207d\"}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"@timestamp\":\"2020-05-03T15:09:59.136+02:00\",\"@version\":1,\"message\":\"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"@timestamp\":\"2020-05-03T15:09:38.255+02:00\",\"@version\":1,\"message\":\"The following profiles are active: default\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.408Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"@timestamp\":\"2020-05-03T15:25:04.709+02:00\",\"@version\":1,\"message\":\"countByDepartmentId called, departmentId: 2\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-3\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"2391430ee2ce38cb\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"2391430ee2ce38cb\",\"X-B3-ParentSpanId\":\"0964d2e2fba9f7e5\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"0964d2e2fba9f7e5\"}\r","@version":"1","@timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}

Now i am more confused. The problem is with logstash or elasticsearch or kibana that I am not able to see any information in kibana?

Answer 1

As discussed in the chat. The problem was the pattern of the index name.

Changing the pattern to logback-* worked perfectly.