CODEWITHSUNDEEP
logstashlogstash-grok
Satya
Satya

Reputation: 1037

Logstash grok config error - while parsing file name

I am new to logstash and grok and I am trying to parse AWS ECS logs in an S3 bucket in the following format -

File Name - my-logs-s3-bucket/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz

Sample Lines - 

2019-05-09T16:16:16.983Z   JBoss Bootstrap Environment
2019-05-09T16:16:16.983Z   JBOSS_HOME: /app/jboss
2019-05-09T16:16:16.983Z   JAVA_OPTS:  -server -XX:+UseCompressedOops  -Djboss.server.log.dir=/var/log/jboss -Xms128m -Xmx4096m

And logstash.conf

input {
  s3 {
    region => "us-east-1"
    bucket => "my-logs-s3-bucket"
    interval => "7200"
  }
}

filter {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
  }
  date {
    match => ["tstamp", "ISO8601"]
  }
  mutate {
    remove_field => ["tstamp"]
    add_field => {
      "file" => "%{[@metadata][s3][key]}"
    }
    ######### NEED HELP HERE - START #########
    #grok {
    #  match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    #}
    ######### NEED HELP HERE - END #########
  }
}

output {
 stdout { codec => rubydebug {
    #metadata => true
  }
 }
}

I am able to see all the logs parsed and the file name extracted when I run logstash using the above configuration and the file name from the output looks like below -

"file" => "myapp-logs/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz",

I am trying to use grok to extract the file name as either ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842 or 8ff0e472-c76f-4f61-a363-64c2b80aa842 by uncommenting grok config lines between #NEED HELP HERE - START and ending with the below error -

Expected one of #, => at line 21, column 10 (byte 536) after filter {\n grok {\n match => [\"message\", \"%{TIMESTAMP_ISO8601:tstamp}\"]\n }\n date {\n match => [\"tstamp\", \"ISO8601\"]\n }\n mutate {\n #remove_field => [\"tstamp\"]\n add_field => {\n \"file\" => \"%{[@metadata][s3][key]}\"\n }\n grok ", :

I am not sure where I am going wrong with this. Please advice.

Upvotes: 0

Views: 117

Answers (1)

leandrojmp
leandrojmp

Reputation: 7463

Your grok filter was inside the mutate filter, try the following.

filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
    }
    date {
        match => ["tstamp", "ISO8601"]
    }
    mutate {
        remove_field => ["tstamp"]
        add_field => { "file" => "%{[@metadata][s3][key]}" }
    }
    grok {
        match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    }
}

Upvotes: 1

Related Questions