CODEWITHSUNDEEP
file-iooutputlogstashgrok
Akk
Akk

Reputation: 406

File input not working for logstash

I get the correct output when I use stdin as an input stream. But whenever I use the file as an input, the output freezes after the following message. 

"Using milestone 2 input plugin 'file'. This plugin should be stable but if you see strange behavior, please let us know."

Here is my config file.

input {
  file {
    path => ["c:/users/a/b/c/logstash-1.4.1/bin/logs/logfile.log"]
    start_position => beginning
  }
}

filter {
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{MY_DATE:my_date}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{DATE:date}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{TIME:time}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{LOG_LEVEL:log_level}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{SERVER:server}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{CLASS_NAME:class_name}"]
        }

}

output {
  stdout { codec => rubydebug }
  elasticsearch { host => localhost }
 }

Is my file path in the wrong format?

Upvotes: 3

Views: 7808

Answers (3)

user5494513
user5494513

Reputation: 1

I am using logstash-1.5.4 on mac. I encountered a similar problem and have resolved it by explicitly setting the sincedb_path in conf file.

sincedb_path => "your sincedb path"

I am not sure if this solution would solve your problem since we are using different version of logstash and different os.

You can run the command with debug mode to see what is happening.

./bin/logstash -f my_logstash.conf --debug

Upvotes: 0

John Petrone
John Petrone

Reputation: 27497

Two issues here:

  1. The sincedb processs to keep track of read location in a file is currently broken in Windows: https://logstash.jira.com/browse/LOGSTASH-429 . There is a patch but others have confirmed that this is still broken as of version 1.4.1
  2. I believe you need quotes around beginning - according to the documentation its "beginning" or "end", defaulting to "end" for start_position. So I suspect it's just ignoring your directive.

start_position

Value can be any of: "beginning", "end"
Default value is "end"

Choose where Logstash starts initially reading files: at the beginning or at the end. The default behavior treats files like live streams and thus starts at the end. If you have old data you want to import, set this to ‘beginning’

The combination is likely causing you to always start at the end of the file.

I can't test under windows right now (no access at work) but wanted to get this out to you.

Upvotes: 4

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

Do you write new logs to your log file?

The start_position option only modifies “first contact” situations where a file is new and not seen before. It is because Logstash will save a sincedb for each file to keep track the current position of monitored log files. So, next time when you restart Logstash, Logstash will start monitor the file based on the sincedb record and the start_position will not work.

So, if you want to import old logs, you have to delete all the .sincedb file before you start logstash and add the start_position option.

Upvotes: 5

Related Questions