CODEWITHSUNDEEP
logstashlogstash-configuration
Arun Mohan
Arun Mohan

Reputation: 978

How to parse multiple log files using logstash?

I have a few log files like below data_log_01.log data_log_02.log data_log_03.log data_log_04.log

Is there any way that I can parse these logs one by one using a single config file in logstash?

Upvotes: 2

Views: 1920

Answers (2)

Abdul
Abdul

Reputation: 1208

In logstatsh you can read from two different microservices logs using the below configuration. It creates two different indices for two different microservices in logstash. You can analyze separate logs of each microservice and you can create a data view based on index format myapp-* to analyze logs aggregated logs.

input {
  file {
    type => "myapp-ms1"
    path => "C:/D_Drive/Dev/ELK/myapp-ms1.log"
    codec => multiline {
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
      negate => "true"
      what => "previous"
    }
  }
  
  file {
    type => "myapp-ms2"
    path => "C:/D_Drive/Dev/ELK/myapp-ms2.log"
    codec => multiline {
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
      negate => "true"
      what => "previous"
    }
  }
}
 
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  } else if [type] == "myapp-ms1" {
    mutate { add_field => { "[@metadata][target_index]" => "myapp-ms1-%{+YYYY.MM}" } }
  } else if [type] == "myapp-ms2" {
    #Index name should always be lower case
    mutate { add_field => { "[@metadata][target_index]" => "myapp-ms2-%{+YYYY.MM}" } }
  }
 
}
 
output {
   
  stdout {
    codec => rubydebug
  }
 
  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["https://localhost:9200/"]
    user => "elastic"
    password => "c8_Hm9ebCcGTB-_5YnZm"
    cacert => "C:/D_Drive/Dev/ELK/elasticsearch-8.6.2/elasticsearch-ssl-http/kibana/elasticsearch-ca.pem"
    index => "%{[@metadata][target_index]}"
  }
}

Upvotes: 1

Phonolog
Phonolog

Reputation: 6511

How about using the file input plugin with a wildcard?

An exmaple configuration could look like this, assuming your log files are located /home/of/your/logs/:

input {
    file {
        path => [
            "/home/of/your/logs/*.log"
            ]
        }
    }
}

The path value has to be an absolute path! You might want to see the docs on using path.

Upvotes: 2

Related Questions