CODEWITHSUNDEEP
logstashgloblogstash-forwarder
Siddharth Trikha
Siddharth Trikha

Reputation: 2506

Using glob on logstash server machine?

We have a separate server for logstash and logs are on a remote machine. We ship these same logs from a remote machine to logstash server using lumberjack's plugin for logstash.

I tried this:

Client config (where logs are present):

input {
        file{
                path => "/home/Desktop/Logstash-Input/**/*_log"
        }
}

output {

        lumberjack {
                hosts => ["xx.xx.xx.xx"]
                port => 4545
                ssl_certificate => "./logstash.pub"
    }

I want to extract fields from my file input's path variable, so that accordingly for different fields values different parsing patterns can be applied.

Eg: Something like this

grok {
    match => ["path", "/home/Desktop/Logstash-Input/(?<server>[^/]+)/(?<logtype>[^/]+)/(?<logdate>[\d]+.[\d]+.[\d]+)/(?<logfilename>.*)_log"]
}

Here server, logtype are directories names which i want in my fields to apply different parsing patterns like:

filter{

   if [server] == "Server2" and [logtype] == "CronLog" {
               grok........
}


if [server] == "Server3" and [logtype] == "CronLog" {
               grok............
}
}

How shall I be able apply the above on my logstash-server config, as file input is on the client machine from which I want to extract fields from path ???

Lumberjack succesfully ships logs to server. I tried applying the grok on client:

grok {
    match => ["path", "/home/Desktop/Logstash-Input/(?<server>[^/]+)/(?<logtype>[^/]+)/(?<logdate>[\d]+.[\d]+.[\d]+)/(?<logfilename>.*)_log"]
}

I checked on client console it adds fields like server, logtype to the logs but on logstsh-server console the fields are not added.

How should I be able to achieve the above????

Upvotes: 0

Views: 133

Answers (1)

Alain Collins
Alain Collins

Reputation: 16362

Two options:

  1. Set the fields when they are originally shipped. The full logstash and logstash-forwarder (aka lumberjack) allow you to do this.
  2. grok the information from the file path, which my documents have in a field called "file". Check your documents to find the actual field name.

Upvotes: 0

Related Questions