ElasticSearch Grok Pattern issue for Custom Log String

Question

I am trying to create a complete GROK pattern on Elasticsearch for the following custom JSON based log:

------------------------DEBUG----------------------------
Date   : 2019-12-26 12:18:21,498 
METHOD NAME:   xyz
{
  "methodName": "SMS_POOL_IN",
  "Tran_Type": "Response",
  "URL": "xyz.abcL",
  "ApiResult": "Success",
  "Date": "2019/12/26 12:18:21",
  "ErrorCode": "00",
  "ErrorReason": "Success",
  "Msisdn": "9999999",
  "CNIC": "99999999",
  "RequestID": "1111",
  "SR_TranID": "2222",
  "Channel": "abc"
} 

but when ever I parse this, i get only timestamp from the grok.

I am using grok debugger to test this. Whenever i use greedydata, i get only first json parameter, rest gets ignored, am i missing something here? how can i make a grok from these logs? Any helping hand would be appreciated

I have created below grok

%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}%{SPACE \s}%{GREEDYDATA.*}

and getting following result with this.

{
  "GREEDYDATA": [
    [
      "------------------------DEBUG----------------------------",
      "Date   : 2019-12-26 12:18:21,498 ",
      "METHOD NAME:   xyz",
      "{",
      ""methodName": "SMS_POOL_IN",",
      ""Tran_Type": "Response",",
      ""URL": "xyz.abcL",",
      ""ApiResult": "Success",",
      ""Date": "2019/12/26 12:18:21",",
      ""ErrorCode": "00",",
      ""ErrorReason": "Success",",
      ""Msisdn": "9999999",",
      ""CNIC": "99999999",",
      ""RequestID": "1111",",
      ""SR_TranID": "2222",",
      ""Channel": "abc"",
      "} ",
      "",
      ""
    ]
  ],
  "SPACE": [
    [
      "\n",
      "\n",
      "\n",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n  ",
      "\n",
      "",
      ""
    ]
  ]
}

I need to show all these json tags segregated as i need them to populated in seperate tags in ELK

Answer 1

I have created the grok myself, the only issue was with the syntax i was trying to build the grok. Below is the correct grok syntax to read above

%{TIMESTAMP_ISO8601:date_time}\s*%{GREEDYDATA:Method}\n%{GREEDYDATA:Bracket}\s*\"methodName\"\:\s\"%{DATA:methodName}\s*\"Tran_Type\"\:\s\"%{DATA:Tran_Type}\s*\"URL\"\:\s\"%{DATA:URL}\s*\"ApiResult\"\:\s\"%{DATA:ApiResult}\s*\"Date\"\:\s\"%{DATA:Date}\s*\"ErrorCode\"\:\s\"%{DATA:ErrorCode}\s*\"ErrorReason\"\:\s\"%{DATA:ErrorReason}\s*\"Msisdn\"\:\s\"%{DATA:Msisdn}\s*\"CNIC\"\:\s\"%{DATA:CNIC}\s*\"RequestID\"\:\s\"%{DATA:RequestID}\s*\"SR_TranID\"\:\s\"%{DATA:SR_TranID}\s*\"Channel\"\:\s\"%{DATA:Channel}\s

First i picked up time stamp, then i picked up everything which is outside my json string in GREEDYDATA, and then i segregated json tags with DATA keyword.

result for above is

{
  "date_time": [
    [
      "2019-12-26 12:18:21,498"
    ]
  ],
  "YEAR": [
    [
      "2019"
    ]
  ],
  "MONTHNUM": [
    [
      "12"
    ]
  ],
  "MONTHDAY": [
    [
      "26"
    ]
  ],
  "HOUR": [
    [
      "12",
      null
    ]
  ],
  "MINUTE": [
    [
      "18",
      null
    ]
  ],
  "SECOND": [
    [
      "21,498"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "Method": [
    [
      "METHOD NAME:   xyz"
    ]
  ],
  "Bracket": [
    [
      "{"
    ]
  ],
  "methodName": [
    [
      "SMS_POOL_IN","
    ]
  ],
  "Tran_Type": [
    [
      "Response","
    ]
  ],
  "URL": [
    [
      "xyz.abcL","
    ]
  ],
  "ApiResult": [
    [
      "Success","
    ]
  ],
  "Date": [
    [
      "2019/12/26 12:18:21","
    ]
  ],
  "ErrorCode": [
    [
      "00","
    ]
  ],
  "ErrorReason": [
    [
      "Success","
    ]
  ],
  "Msisdn": [
    [
      "9999999","
    ]
  ],
  "CNIC": [
    [
      "99999999","
    ]
  ],
  "RequestID": [
    [
      "1111","
    ]
  ],
  "SR_TranID": [
    [
      "2222","
    ]
  ],
  "Channel": [
    [
      "abc""
    ]
  ]
}

Answer 2

I assume you wanted to separate out first 3 lines into 3 separate fields and rest of the JSON string object into another field.

I copied your input text from here so every line ended with \n. So this was my pattern match element.

Let me know if the output needed more parsing etc.

my pipeline configuration that has grok pattern to parse the input

input {                                                                                                                            
           http {

           }

   }

  filter {

          grok {
                  match  => { "message" => "(?<debug-string>[^\n]+)\n(?<date-string>[^\n]+)\n(?<method-name>[^\n]+)\n%{GREEDYDATA:RestOfIt}" }
          }
         mutate {
                 remove_field => ["headers", "host", "@timestamp", "@version"]
         }

  }

  output {
          stdout {

          }
}

The OUTPUT

{ "message" => "------------------------DEBUG----------------------------\nDate : 2019-12-26 12:18:21,498 \nMETHOD NAME: xyz\n{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\",\n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12/26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\",\n \"Channel\": \"abc\"\n} ",

"date-string" => "Date : 2019-12-26 12:18:21,498 ",

"method-name" => "METHOD NAME: xyz",

"RestOfIt" => "{\n \"methodName\": \"SMS_POOL_IN\",\n \"Tran_Type\": \"Response\",\n \"URL\": \"xyz.abcL\",\n \"ApiResult\": \"Success\",\n \"Date\": \"2019/12/26 12:18:21\",\n \"ErrorCode\": \"00\",\n \"ErrorReason\": \"Success\",\n \"Msisdn\": \"9999999\",\n \"CNIC\": \"99999999\",\n \"RequestID\": \"1111\",\n \"SR_TranID\": \"2222\",\n \"Channel\": \"abc\"\n} ",

"debug-string" => "------------------------DEBUG----------------------------" }