CODEWITHSUNDEEP
elasticsearchamazon-s3logstashkibana-4amazon-cloudtrail
KarlH
KarlH

Reputation: 23

Linux ELK server reading cloudtrail logs - troubleshooting issues

Have built a ELK server running on RHEL 7.2 in AWS. The plan is to have it ingest cloudtrail logs from a S3 bucket and then do magic at the kibana front end but it's not working and I've lost days of my life trying to work it out hence why I'm reaching out for some help. I can see from my logstash.log that it's reading the files in the S3 bucket but that's about it. Nothing else appears to be happening.

my setup:

RHEL 7.2
kibana-4.5.0-1.x86_64
logstash-2.3.2-1.noarch
elasticsearch-2.3.3-1.noarch
nginx-1.6.3-8.el7.x86_64 (reverse proxy kibana to port 80)


This is what my /etc/logstash/conf.d looks like :
-rw-r--r-- 1 root root 574 May 31 14:55 02-cloudtrail-input.conf
-rw-r--r-- 1 root root 432 May 31 15:04 30-elasticsearch-output.conf

root@elk conf.d]# cat *

input {   
    s3 
    bucket => "xyz..cloudtrail"
        access_key_id => 'XYZ'
        secret_access_key => 'ABC'
        delete => false
        codec => "cloudtrail"
        prefix => "cloudtrail/AWSLogs/xxxxx/CloudTrail/ap-southeast-2/2016/"
        type => "cloudtrail"
        interval => 10 # seconds
        region => "ap-southeast-2"
        sincedb_path => "/data/logstash/cloudtrail/db/sincedb"
    }
}
output {
        #stdout {}
        stdout { codec => rubydebug }
    elasticsearch {
        hosts => "localhost:9200"
        sniffing => true
        #codec => "cloudtrail"
        #index => "cloudtrail"
        index => "logstash-%{+YYYY.MM.dd}"
        #index => "%{[@metadata][cloudtrail]}-%{+YYYY.MM.dd}"
        #index => "cloudtrail-%{+YYYY.MM.dd}"
        action => create
        manage_template => false
    workers => 4
    }
}

installed plugins :
logstash-codec-cloudtrail
logstash-input-s3
logstash-output-s3

I don't know how a index should be defined in the logstash output but I can search for it on the kibana front end and select one of 3 time-frame options but don't know what that means. Should I have sniffing = true ? Should I have action defined => create ? Should I have the cloudtrail codec defined in both the logstash input and output ?

Can anyone tell me what it means when it says "Plugin not defined in namespace, checking for plugin file" ? sounds to me like it can't find the plugin even though all required plugins are installed (I think)

Here is the output of when I start logstash manually... but it doesn't mean much to me..

Reading config file {:config_file=>"/etc/logstash/conf.d/02-cloudtrail-input.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
Reading config file {:config_file=>"/etc/logstash/conf.d/30-elasticsearch-output.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"s3", :path=>"logstash/inputs/s3", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"cloudtrail", :path=>"logstash/codecs/cloudtrail", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Codecs::CloudTrail/@spool_size = 50 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@bucket = "abcdbase-cloudtrail" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@access_key_id = "XYZ" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@secret_access_key = "ABC" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@delete = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@codec = <LogStash::Codecs::CloudTrail spool_size=>50> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@prefix = "abcdbase-trail/AWSLogs/554658506446/CloudTrail/ap-southeast-2/2016/" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@type = "cloudtrail" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@interval = 10 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@region = "ap-southeast-2" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@sincedb_path = "/data/logstash/cloudtrail/db/sincedb" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@use_ssl = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@credentials = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_to_bucket = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_add_prefix = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_to_dir = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@exclude_pattern = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@temporary_directory = "/tmp/logstash" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"elasticsearch", :path=>"logstash/outputs/elasticsearch", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}

what does it mean when the logstash.log keeps saying method=>"list_new_files", does that mean it's busy re-reading the S3 bucket (every time logstash is stopped/started) ?

I know that's a lot of questions but thought I would try for some help before throwing this setup in the bin.

thanks

Upvotes: 0

Views: 503

Answers (1)

Dilip Panwar
Dilip Panwar

Reputation: 1

Below config is working for me.

input {
s3 {
  access_key_id => "xxxx"
  bucket => "my-bucket"
  region => "us-east-1"
  secret_access_key => "xxx"
  prefix => "your prefix"
  type => "s3"
  add_field => { source => gzfiles }
  codec => cloudtrail {}
}
}

output {
  elasticsearch {
        hosts => localhost
    }
}

I installed the plugins mentioned below-

logstash-codec-cloudtrail logstash-input-s3

logstash-output-elasticsearc

For more plugin details, you may fire plugin list command-

/opt/logstash/bin/plugin list

Upvotes: 0

Related Questions