CODEWITHSUNDEEP
logstash
user2443476
user2443476

Reputation: 1975

Use regex as input file path in Logstash

I would like to parse a directory of logs files with logstash. When the logs are formatted like this :

server-20140604.log
server-20140603.log
server-20140602.log

There is no problem, I am using globs like this :

input {
 file {
   path=>["D:/*.log"]
 }
}

But my logs are formatted like this :

server.log
server.log.1
server.log.2
client.log
client.log.1
client.log.2

So I would like to know how to tell to logstash to parse in the folder all the files starting with "server" expression in their names. I really need to do it like that, because I have other files in the folder (i.e client logs) that I don't want to parse but also cannot remove from the folder.

Upvotes: 1

Views: 2342

Answers (1)

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

With this configuration I can only parse all the log files start with prefix server.

input {
        file {
                path => ["D:/server*"]
        }
}

output {
    stdout {
                codec => rubydebug
        }
}

I think the possible problem you have meet is the start_position config. It means that where does logstash start to read the logs. Please refer to here. Remember this option only modifies first contact situations where a file is new and not seen before. If a file has already been seen before, this option has no effect.

When you stop logstash, logstash will save a .sincedb* in your home directory. Next time you start it, logstash will start read the file according to .sindb*. If you do not input new logs to server.log, logstash will never parse the old logs.

What you can try to do is delete all the .sincedb before you start logstash and add start_posistion to your config. In your comment you have say if you overwrite the server.log logstash can parse the file from beginning, it is because logstash detect it as a new file and the .sincedb* do not save any information about this file. So logstash will parse it! You can try to find out your .sincedb and try to delete it.

Upvotes: 1

Related Questions