CODEWITHSUNDEEP
logstash
Peter McEvoy
Peter McEvoy

Reputation: 2926

In logstash, do I need separate file inputs for logically different application logs?

(Logstash 1.4.2 on Windows)

In our system, a "product" is a high level grouping of related web applications. Each web application is configured to write a dedicated log file, named after the application name (eg MyProduct.ApplicationA.log and MyProduct.ApplicationB.log). All web applications for a given product write their log files to the same folder (c:\Logs\MyProduct\; c:\Logs\MyOtherProduct).

I need to set up logstash to monitor all log files for all applications for all products. I had hoped to use:

input {
    file {
        path => "c:/Logs/**/*.log"
        exclude => ["Info.*", "Warn.*", "Error.*"]
        sincedb_path => "c:/logstash/.sincedb"
        sincedb_write_interval => 1
    }
}

On first run, I can see lots of output going to stdout output, which I presume is what the docs refer to as "first contact".

Once all the log files (from more than one application) have been initially parsed, if applications generate log entries, they appears to be picked up and output. All is well.

However, if I restart logstash, ALL the logfiles seem to be parsed again - as if sincedb is not honoured. I have looked at the other SO questions detailing similar experience of duplicates and reparsing (eg logstash + elasticsearch : reloads the same data), however I believe that I have extra information that may indicate that I am actually using the file input incorrectly.

If I instead setup multiple file inputs like so:

file {
    path => "c:/Logs/MyProduct/MyProduct.ApplicationA.log"
    exclude => ["Info.*", "Warn.*", "Error.*"]
    sincedb_path => "c:/logstash/.sincedb_A"
    sincedb_write_interval => 1
}
file {
    path => "c:/Logs/MyProduct/MyProduct.ApplicationB.log"
    exclude => ["Info.*", "Warn.*", "Error.*"]
    sincedb_path => "c:/logstash/.sincedb_B"
    sincedb_write_interval => 1
}

Then restarts of logstash do not reparse existing files and do honour the sincedb for the logical grouping. This leads me to believe that perhaps I have been thinking about the file input in the wrong way: will I have to configure an individual file inputs for each application?

(Looking at the content of sincedb, there is only ever a single line eg

0 0 2 661042

and it becomes obvious that multiple files cannot be tracked)

Am I missing something that would allow me to have a generic globular style global declaration, without needing to do individual per-application configuration?

Upvotes: 3

Views: 581

Answers (1)

user4383461
user4383461

Reputation:

Looks like you're running into a known sincedb bug on Windows

Your workaround of adding a file {} block with separate sincedb_path for each file is probably the best solution until the bug is fixed.

Upvotes: 3

Related Questions