CODEWITHSUNDEEP
windowselasticsearchlogstashelastic-stack
Ray
Ray

Reputation: 4947

How can I debug logstash even when configuration is OK, on Windows?

I have the following configuration for my logstash importing a few CSV files:

input {
  file {
    path => [
        "C:\Data\Archive_ATS_L1\2016-10-08-00-00_to_2016-10-09-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-09-00-00_to_2016-10-10-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-10-00-00_to_2016-10-11-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-11-00-00_to_2016-10-12-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-12-00-00_to_2016-10-13-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-13-00-00_to_2016-10-14-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-14-00-00_to_2016-10-15-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv"
    ]
    start_position => "beginning"    
  }
}
filter {
  csv {
      separator => ","
      columns => ["MessageCode","SourceGuid","DateTimeGenerated","Code1","Code2","Code3","Code4","LanguageCode", "AlarmSeverity", "Message", "Guid1", "Guid2", "Guid3", "Guid4", "MessageOrigin", "RequestId", "Bool1", "Bool2", "Bool3", "Bool4", "Bool5", "Bool6", "Bool7", "Bool8", "Code5", "Code6", "Bool9", "Bool10", "Bool11", "Code7"]
  }
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "S2K"
        workers => 1
    }
    stdout {}
}

I launch logstash with this command line:

logstash.bat –f ..\conf\logstash.conf --verbose

Usually I see the data that's being imported into Elasticsearch in the console. But all I get this time is one line that says "Pipeline main started" and it stays like that.

How can I check from logstash if data was imported? I tried using Elasticsearch by running: curl http://localhost:9200/_aliases

This usually gives the list of indices. But the index I have in this config (called S2K) does not get listed.

I'm new to ELK so how can I check if logstash is doing it's job? Please note that I'm using Windows 7.

Upvotes: 5

Views: 23425

Answers (3)

Dan Griffiths
Dan Griffiths

Reputation: 321

Stdout Ruby Debug is your friend here.

This will output everything to screen so you'll need to push the screen output to a file (example code at the bottom)

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html

What goes in the .conf file within the output section

output { stdout { codec => rubydebug } }

This is the example of how you would run the conf and push the screen output to another file for debugging.

logstash -r -f yourconfig.conf > debugfile.out

Just change yourconfig.conf and debugfile.out for whatever names you want and please remember to remove the rubydebug codec from your conf file when done debugging!

Upvotes: 2

Anton
Anton

Reputation: 730

To debug logstash you need to do two things: add stdout in config, and run logstash in a proper way.

1 step: Add this config in your logstash conf file (ex.: /etc/logstash/conf.d/config.conf)

output {
  stdout {
    codec => rubydebug {
      metadata => true # Here, we will print metadata in console
    }
  }
}

2 step: Run logstash to see output with command

sudo /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/config.conf

And you will get something like this:

{
            "log" => {
        "file" => {
            "path" => "***\\spring.log"
        }
    },
        "appName" => "my-service",
      "@metadata" => {
        "ip_address" => "***",
              "type" => "_doc",
              "beat" => "filebeat",
           "version" => "7.12.0"
    },
      "log_level" => "INFO",
     "serverName" => "***",
            "pid" => "6236",
         "thread" => "main",
        "message" => "***",
    "serviceName" => "***",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "input" => {
        "type" => "log"
    },
     "@timestamp" => 2021-01-03T10:22:07.644Z,
       "@version" => "1",
          "class" => "***"
}

Finally, after debug you can run it like sudo systemctl start logstash

Hope, it would help you, this approach helped me to save my time

Upvotes: 8

Dan Griffiths
Dan Griffiths

Reputation: 321

You may be able to use the line codec to change the charset of how the line is read (default is UTF-8) instead of having to change the files themselves

Upvotes: 0

Related Questions